Jul 01 20:46:06.967175 ip-10-0-138-219 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 20:46:07.473953 ip-10-0-138-219 systemd[1]: Started Kubernetes Kubelet.
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.406870    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413337    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413350    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413355    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413360    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413363    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413367    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413370    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:07.483689 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413374    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413378    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413382    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413387    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413390    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413395    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413399    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413403    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413407    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413411    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413415    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413419    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413423    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413427    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413431    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413435    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413439    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413443    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413446    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413450    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:07.484691 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413454    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413458    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413462    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413466    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413470    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413474    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413478    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413481    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413485    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413490    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413494    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413497    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413502    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413506    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413510    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413513    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413517    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413521    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413527    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413531    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:07.485317 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413537    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413541    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413545    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413549    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413553    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413556    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413561    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413564    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413569    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413572    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413575    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413578    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413582    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413585    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413589    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413593    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413597    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413601    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413605    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413609    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:07.485979 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413613    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413617    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413621    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413625    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413629    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413633    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413637    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413640    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413644    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413647    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413651    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413654    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413658    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413664    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413668    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413673    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413677    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413681    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413684    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:07.486583 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413688    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413696    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413701    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413705    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413710    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413714    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413718    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413723    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413728    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413732    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413736    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413740    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.413744    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416701    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416709    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416712    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416716    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416719    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416723    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416726    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:07.487101 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416728    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416731    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416733    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416736    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416739    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416742    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416745    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416748    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416750    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416753    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416756    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416759    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416762    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416765    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416769    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416773    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416776    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416780    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416782    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:07.487664 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416785    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416788    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416790    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416793    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416796    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416798    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416801    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416804    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416806    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416809    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416812    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416815    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416817    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416820    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416822    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416824    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416827    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416829    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416832    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416835    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:07.488221 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416837    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416840    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416842    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416845    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416848    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416874    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416880    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416883    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416886    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416888    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416891    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416894    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416897    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416900    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416902    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416905    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416908    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416910    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416913    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:07.488836 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416916    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416918    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416920    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416923    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416927    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416930    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416933    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416935    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416938    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416940    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416943    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416949    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416951    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416954    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416957    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416959    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416964    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416967    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416970    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416973    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:07.489371 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416976    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416979    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416982    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416985    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416988    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416990    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416993    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416995    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.416998    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.417001    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.417003    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.417005    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.417008    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.417010    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418429    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418441    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418446    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418449    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418452    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:07.489951 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418456    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418459    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418462    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418466    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418469    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418472    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418475    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418478    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418481    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418483    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418486    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418489    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418492    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418494    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418497    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418499    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418501    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418504    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418509    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418511    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:07.490455 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418514    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418517    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418519    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418522    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418525    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418527    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418530    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418538    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418541    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418544    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418546    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418562    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418566    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418569    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418572    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418575    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418578    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418580    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418583    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418585    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:07.491253 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418588    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418590    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418593    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418597    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418600    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418604    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418608    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418611    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418614    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418617    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418621    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418623    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418626    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418628    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418630    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418633    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418636    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418639    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418642    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:07.492159 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418646    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418648    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418650    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418653    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418655    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418658    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418660    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418663    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418665    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418668    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418671    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418673    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418676    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418678    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418680    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418683    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418685    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418688    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418692    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418694    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:07.493297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418697    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418699    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418702    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418705    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418708    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418710    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418713    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418715    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418718    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418720    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418723    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418725    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418729    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418731    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.418733    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418794    2573 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418801    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418806    2573 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418810    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418816    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418821    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 20:46:07.494333 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418827    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418832    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418837    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418841    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418844    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418847    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418866    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418869    2573 flags.go:64] FLAG: --cgroup-root=""
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418872    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418875    2573 flags.go:64] FLAG: --client-ca-file=""
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418879    2573 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418882    2573 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418886    2573 flags.go:64] FLAG: --cluster-domain=""
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418889    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418893    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418896    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418899    2573 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418903    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418906    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418909    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418912    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418915    2573 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418918    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418922    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 20:46:07.495417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418925    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418928    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418932    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418935    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418938    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418940    2573 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418943    2573 flags.go:64] FLAG: --enable-server="true"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418946    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418950    2573 flags.go:64] FLAG: --event-burst="100"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418953    2573 flags.go:64] FLAG: --event-qps="50"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418956    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418959    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418962    2573 flags.go:64] FLAG: --eviction-hard=""
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418973    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418975    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418981    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418985    2573 flags.go:64] FLAG: --eviction-soft=""
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418988    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418990    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418994    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.418996    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419000    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419004    2573 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419006    2573 flags.go:64] FLAG: --feature-gates=""
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419010    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 20:46:07.496540 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419013    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419016    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419019    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419022    2573 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419025    2573 flags.go:64] FLAG: --help="false"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419027    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-138-219.ec2.internal"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419032    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419034    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419037    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419040    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419044    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419046    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419049    2573 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419052    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419055    2573 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419057    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419060    2573 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419063    2573 flags.go:64] FLAG: --kube-reserved=""
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419066    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419069    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419072    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419074    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419077    2573 flags.go:64] FLAG: --lock-file=""
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419080    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 20:46:07.497841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419083    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419085    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419090    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419094    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419096    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419101    2573 flags.go:64] FLAG: --logging-format="text"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419104    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419107    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419110    2573 flags.go:64] FLAG: --manifest-url=""
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419112    2573 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419119    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419122    2573 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419125    2573 flags.go:64] FLAG: --max-pods="110"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419128    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419133    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419136    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419138    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419141    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419144    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419147    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419153    2573 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419156    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419159    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 20:46:07.501383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419162    2573 flags.go:64] FLAG: --pod-cidr=""
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419165    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419170    2573 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419172    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419175    2573 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419178    2573 flags.go:64] FLAG: --port="10250"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419181    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419184    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0870ebe16e7e7e5df"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419187    2573 flags.go:64] FLAG: --qos-reserved=""
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419190    2573 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419192    2573 flags.go:64] FLAG: --register-node="true"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419196    2573 flags.go:64] FLAG: --register-with-taints=""
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419199    2573 flags.go:64] FLAG: --registry-burst="10"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419203    2573 flags.go:64] FLAG: --registry-qps="5"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419207    2573 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419209    2573 flags.go:64] FLAG: --reserved-memory=""
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419213    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419216    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419219    2573 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419221    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419224    2573 flags.go:64] FLAG: --runonce="false"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419227    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419230    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419233    2573 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419237    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 20:46:07.502534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419240    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419243    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419245    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419248    2573 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419251    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419254    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419257    2573 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419259    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419262    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419265    2573 flags.go:64] FLAG: --system-cgroups=""
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419267    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419273    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419276    2573 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419278    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419283    2573 flags.go:64] FLAG: --tls-min-version=""
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419285    2573 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419288    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419291    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419293    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419296    2573 flags.go:64] FLAG: --v="2"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419300    2573 flags.go:64] FLAG: --version="false"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419305    2573 flags.go:64] FLAG: --vmodule=""
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419309    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419312    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419421    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:07.504703 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419424    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419428    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419430    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419433    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419436    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419438    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419443    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419445    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419448    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419451    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419453    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419456    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419458    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419460    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419463    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419466    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419468    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419470    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419473    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419475    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:07.505722 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419478    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419481    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419483    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419486    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419489    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419491    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419493    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419496    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419500    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419504    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419506    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419509    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419511    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419514    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419516    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419519    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419521    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419524    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419528    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419530    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:07.506745 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419533    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419535    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419538    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419541    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419543    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419545    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419548    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419551    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419553    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419555    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419558    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419560    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419563    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419566    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419568    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419570    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419573    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419577    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419580    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:07.507612 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419583    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419588    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419591    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419594    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419596    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419599    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419601    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419605    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419608    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419611    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419614    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419618    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419621    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419623    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419626    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419629    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419631    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419634    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419637    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:07.654163 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419639    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419642    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419644    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419647    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419649    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419652    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419654    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419657    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419659    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419662    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419664    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419668    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419671    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419674    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419678    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419681    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419683    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419686    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419689    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:07.686206 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.419691    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.419696    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.426145    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.426159    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.426178    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.426184    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426236    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426240    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426243    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426247    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426249    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426255    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426257    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426260    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:07.687529 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426263    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426266    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426268    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426272    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426276    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426279    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426282    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426285    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426288    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426293    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426296    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426299    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426301    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426304    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426306    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426309    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426312    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426315    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426317    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426320    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:07.688142 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426322    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426326    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426328    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426331    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426334    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426336    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426339    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426342    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426344    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426348    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426350    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426353    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426355    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426358    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426361    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426363    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426366    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426368    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426371    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426373    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:07.689202 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426376    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426380    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426382    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426385    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426387    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426389    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426392    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426395    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426398    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426400    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426402    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426405    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426408    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426410    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426413    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426416    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426418    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426421    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426423    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426426    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:07.690382 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426428    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426432    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426434    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426437    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426439    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426442    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426444    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426448    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426452    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426454    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426457    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426460    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426463    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426467    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426470    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426472    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426475    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426478    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426481    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:07.691183 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426483    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426486    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426488    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426491    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426493    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426497    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426501    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426503    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426507    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426509    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426512    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426515    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.426519    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426627    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:07.691852 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426631    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426635    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426639    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426642    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426646    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426649    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426651    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426654    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426657    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426659    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426662    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426664    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426669    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426671    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426674    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426678    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426681    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426684    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426686    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:07.692437 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426689    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426692    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426694    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426697    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426700    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426702    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426705    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426707    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426710    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426713    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426716    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426718    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426720    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426723    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426727    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426729    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426732    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426734    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426737    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426739    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:07.693297 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426742    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426745    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426747    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426750    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426752    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426757    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426759    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426762    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426765    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426768    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426770    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426773    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426775    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426778    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426780    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426783    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426785    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426788    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426790    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:07.694062 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426793    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426795    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426801    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426804    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426806    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426809    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426811    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426815    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426818    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426820    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426822    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426825    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426828    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426830    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426833    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426835    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426838    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426840    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426844    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426846    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:07.694984 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426849    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426868    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426871    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426874    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426877    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426879    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426882    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426885    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426887    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426890    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426892    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426895    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426898    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426901    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426904    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426907    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426910    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426913    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426915    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:07.695683 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:07.426919    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.426924    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.427008    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.429462    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.430909    2573 server.go:1019] "Starting client certificate rotation"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.431006    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.431062    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.433552    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.434666    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.443349    2573 log.go:25] "Validated CRI v1 runtime API"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.445373    2573 log.go:25] "Validated CRI v1 image API"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.446186    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 20:46:07.696433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.448780    2573 fs.go:135] Filesystem UUIDs: map[07d3c57a-42c9-442b-b8c8-717663609abe:/dev/nvme0n1p4 792a821d-a0f0-47b9-8852-ee427f3c193b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 01 20:46:07.696870 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.448794    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 20:46:07.875950 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.453572    2573 manager.go:216] Machine: {Timestamp:2026-07-01 20:46:07.452345204 +0000 UTC m=+0.363679822 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100642 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec268d2f308ec32c36ac5bc26c1ae269 SystemUUID:ec268d2f-308e-c32c-36ac-5bc26c1ae269 BootID:0986affd-569b-4a6a-8444-ffbfbc0ac80e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:74:47:64:a9:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:74:47:64:a9:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:7b:92:7c:14:8f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 20:46:07.875950 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.453666    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 20:46:07.875950 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.453768    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 20:46:07.875950 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.457095    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 20:46:07.875950 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.457117    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 20:46:07.875950 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.457910    2573 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 20:46:07.875950 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.457921    2573 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 20:46:07.875950 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.457929    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.458588    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.459445    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.460534    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.462484    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.462583    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.462604    2573 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.462613    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.462623    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.462632    2573 kubelet.go:398] "Adding apiserver pod source"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.462639    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.462916    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.462929    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.466633    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468067    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468094    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468682    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468694    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468699    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468705    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468713    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468719    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468802    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468807    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468814    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468819    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468827    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.468834    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.470233    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.470240    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.473103    2573 server.go:1268] "Started kubelet"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.473181    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.473201    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.473255    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.478601    2573 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.480667    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.482155    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.482176    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.482302    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 20:46:07.876400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.485606    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.490735    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.490809    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.490734    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491303    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491304    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491322    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491405    2573 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491415    2573 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491486    2573 factory.go:55] Registering systemd factory
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491528    2573 factory.go:223] Registration of the systemd container factory successfully
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491722    2573 factory.go:153] Registering CRI-O factory
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491735    2573 factory.go:223] Registration of the crio container factory successfully
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491779    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491801    2573 factory.go:103] Registering Raw factory
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.491814    2573 manager.go:1196] Started watching for new ooms in manager
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.492137    2573 manager.go:349] Starting recovery of all containers
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.492833    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.492848    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-219.ec2.internal.18be455cd3817113  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-219.ec2.internal,UID:ip-10-0-138-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-219.ec2.internal,},FirstTimestamp:2026-07-01 20:46:07.473078547 +0000 UTC m=+0.384413163,LastTimestamp:2026-07-01 20:46:07.473078547 +0000 UTC m=+0.384413163,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-219.ec2.internal,}"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.502650    2573 manager.go:354] Recovery completed
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.506563    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.508275    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.508306    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.508320    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.509694    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.510211    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.510221    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.510223    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.510240    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.511952    2573 policy_none.go:49] "None policy: Start"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.511974    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.511988    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.512748    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.512779    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.877581 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.512792    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.513225    2573 policy_none.go:47] "Start"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.515709    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9d7dq"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.528980    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9d7dq"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.540495    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-219.ec2.internal.18be455cd59ab1c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-219.ec2.internal,UID:ip-10-0-138-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-219.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-219.ec2.internal,},FirstTimestamp:2026-07-01 20:46:07.508287943 +0000 UTC m=+0.419622559,LastTimestamp:2026-07-01 20:46:07.508287943 +0000 UTC m=+0.419622559,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-219.ec2.internal,}"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.551504    2573 manager.go:337] "Starting Device Plugin manager"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.551632    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.551642    2573 server.go:87] "Starting device plugin registration server"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.551837    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.551846    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.551999    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.552099    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.552108    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.552534    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.552579    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.583728    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.584876    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.584912    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.584945    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.584983    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.587551    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.652692    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.653490    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.653510    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.653522    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.653551    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-219.ec2.internal"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.663483    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-219.ec2.internal"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.663499    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-219.ec2.internal\": node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.689148    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal"]
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.689203    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.690623    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.878620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.690649    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.690659    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.692506    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.692602    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.692634    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.693226    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.693236    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.693253    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.693261    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.693267    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.693277    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.694843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.694872    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.695574    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.695599    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.695610    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.705958    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.729569    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-219.ec2.internal\" not found" node="ip-10-0-138-219.ec2.internal"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.734321    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-219.ec2.internal\" not found" node="ip-10-0-138-219.ec2.internal"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.792491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5835e552044786854dabbf9939043124-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"5835e552044786854dabbf9939043124\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.792516    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5835e552044786854dabbf9939043124-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"5835e552044786854dabbf9939043124\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:07.879520 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.807073    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:08.010225 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.893366    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5835e552044786854dabbf9939043124-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"5835e552044786854dabbf9939043124\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.010225 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.893424    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5835e552044786854dabbf9939043124-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"5835e552044786854dabbf9939043124\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.010225 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.893462    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5835e552044786854dabbf9939043124-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"5835e552044786854dabbf9939043124\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.010225 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.893497    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5835e552044786854dabbf9939043124-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"5835e552044786854dabbf9939043124\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.010225 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.893499    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75bb94aa8471c122d1bbd3ff2e95e5c0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-219.ec2.internal\" (UID: \"75bb94aa8471c122d1bbd3ff2e95e5c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.010225 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:07.907501    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:08.010225 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.994599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75bb94aa8471c122d1bbd3ff2e95e5c0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-219.ec2.internal\" (UID: \"75bb94aa8471c122d1bbd3ff2e95e5c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.010225 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:07.994653    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75bb94aa8471c122d1bbd3ff2e95e5c0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-219.ec2.internal\" (UID: \"75bb94aa8471c122d1bbd3ff2e95e5c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.011032 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:08.010883    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:08.123050 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.033276    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.123050 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.047718    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.123050 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:08.111308    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:08.239382 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:08.211924    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:08.365560 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:08.313029    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:08.520112 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:08.413793    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:08.520112 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.430302    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 20:46:08.520112 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.430429    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 20:46:08.520112 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:08.514252    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:08.621485 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.532807    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 20:41:08 +0000 UTC" deadline="2028-04-01 13:08:47.826657294 +0000 UTC"
Jul 01 20:46:08.621485 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.532843    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15352h22m39.293817963s"
Jul 01 20:46:08.621485 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.539034    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:08.621485 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.539556    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:08.621485 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.565875    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:08.621485 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.567232    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:08.621485 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.567259    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:08.621485 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.567269    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:08.621485 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:08.614442    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found"
Jul 01 20:46:08.760899 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.643406    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:08.760899 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:08.689335    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5835e552044786854dabbf9939043124.slice/crio-87a3ed0be67da9da1ec00f46a3bafbc42fb460f95cfefdd2d1e3be132e3e8078 WatchSource:0}: Error finding container 87a3ed0be67da9da1ec00f46a3bafbc42fb460f95cfefdd2d1e3be132e3e8078: Status 404 returned error can't find the container with id 87a3ed0be67da9da1ec00f46a3bafbc42fb460f95cfefdd2d1e3be132e3e8078
Jul 01 20:46:08.760899 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:08.689676    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bb94aa8471c122d1bbd3ff2e95e5c0.slice/crio-fa65f54a5e881bc834ab9781175c073ead4673ce5e8c3b120192d381b01b875a WatchSource:0}: Error finding container fa65f54a5e881bc834ab9781175c073ead4673ce5e8c3b120192d381b01b875a: Status 404 returned error can't find the container with id fa65f54a5e881bc834ab9781175c073ead4673ce5e8c3b120192d381b01b875a
Jul 01 20:46:08.760899 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.691994    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.760899 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.694265    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:46:08.760899 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.704660    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 20:46:08.760899 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.705774    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal"
Jul 01 20:46:08.760899 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:08.719322    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 20:46:09.452302 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.452274    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:09.466937 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.466911    2573 apiserver.go:52] "Watching apiserver"
Jul 01 20:46:09.476325 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.476301    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 20:46:09.478700 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.478672    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zwrwf","kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal","openshift-cluster-node-tuning-operator/tuned-wjlv6","openshift-dns/node-resolver-llqwv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal","openshift-multus/multus-85vx9","openshift-multus/multus-additional-cni-plugins-xb4ww","openshift-multus/network-metrics-daemon-sqdk2","kube-system/konnectivity-agent-n8ht4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9","openshift-image-registry/node-ca-ffv5j","openshift-network-diagnostics/network-check-target-wzfcf","openshift-network-operator/iptables-alerter-h52h5","openshift-ovn-kubernetes/ovnkube-node-lrvqz"]
Jul 01 20:46:09.480704 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.480678    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.483975 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.483949    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.484761 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.484695    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 20:46:09.484851 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.484831    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 20:46:09.485137 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.485117    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 20:46:09.485228 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.485177    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 20:46:09.485692 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.485326    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.485692 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.485371    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f5r6k\""
Jul 01 20:46:09.489079 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.487580    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.489079 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.487679    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 20:46:09.489079 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.487946    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.489079 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.488688    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 20:46:09.489079 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.488700    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p5m62\""
Jul 01 20:46:09.489079 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.488966    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 20:46:09.489438 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.489219    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:09.489499 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.489485    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 20:46:09.492337 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.489601    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:09.492337 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.489669    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4nc2v\""
Jul 01 20:46:09.492337 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.489901    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.492337 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.490956    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 20:46:09.492337 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.491467    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 20:46:09.492337 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.491666    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 20:46:09.492337 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.491852    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x6mpj\""
Jul 01 20:46:09.495500 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.494592    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:09.495500 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.494712    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:09.495500 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.494857    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 20:46:09.495500 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.495101    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 20:46:09.495500 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.495406    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 20:46:09.495799 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.495634    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c8hxm\""
Jul 01 20:46:09.496010 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.495990    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 20:46:09.496295 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.496271    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lnwpp\""
Jul 01 20:46:09.496370 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.496302    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 20:46:09.496451 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.496305    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 20:46:09.496744 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.496722    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.499431 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.499402    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2stvp\""
Jul 01 20:46:09.499928 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.499909    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:09.500415 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.500153    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:09.500415 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.500157    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 20:46:09.500415 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.500411    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:09.500631 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.500482    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:09.500925 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.500878    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 20:46:09.501770 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.501725    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.502951 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.502927    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-var-lib-cni-bin\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.503085 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.502963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-var-lib-openvswitch\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.503085 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.502989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-ovnkube-config\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.503085 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-socket-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.503085 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-cni-dir\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.503085 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-run-k8s-cni-cncf-io\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.503362 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503094    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9vvb\" (UniqueName: \"kubernetes.io/projected/82ea0a51-b205-4ba9-8356-a96baf3f0de6-kube-api-access-z9vvb\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.503362 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-tuned\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.503362 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503145    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.503362 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503169    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26q5\" (UniqueName: \"kubernetes.io/projected/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-kube-api-access-g26q5\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.503362 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503192    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-system-cni-dir\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-os-release\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503430    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82ea0a51-b205-4ba9-8356-a96baf3f0de6-cni-binary-copy\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-systemd\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503495    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-cni-bin\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6zn\" (UniqueName: \"kubernetes.io/projected/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-kube-api-access-4d6zn\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503545    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33310520-cb48-4e1e-b89d-f3373d57e646-host-slash\") pod \"iptables-alerter-h52h5\" (UID: \"33310520-cb48-4e1e-b89d-f3373d57e646\") " pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503567    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-device-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mrc\" (UniqueName: \"kubernetes.io/projected/f4938d98-b1e1-4724-97e2-0e948765bd87-kube-api-access-57mrc\") pod \"node-resolver-llqwv\" (UID: \"f4938d98-b1e1-4724-97e2-0e948765bd87\") " pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503614    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-run-netns\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.503657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-sysctl-d\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503662    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-log-socket\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503685    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-ovn-node-metrics-cert\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503709    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-cnibin\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503732    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-os-release\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-registration-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4938d98-b1e1-4724-97e2-0e948765bd87-tmp-dir\") pod \"node-resolver-llqwv\" (UID: \"f4938d98-b1e1-4724-97e2-0e948765bd87\") " pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-run-multus-certs\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-run-openvswitch\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503906    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-sysconfig\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a404fa55-b2ca-477c-9a5a-23e4e079778a-tmp\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503961    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-kubelet\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.503985    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-run-netns\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.504009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-var-lib-kubelet\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.504423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.504164    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:09.505113 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.504218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-sysctl-conf\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.505113 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.504278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8576212-b59b-4cf3-9043-7744f159a399-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.505113 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.504304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.505113 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.504630    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bmr8d\""
Jul 01 20:46:09.505113 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.504679    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 20:46:09.505113 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.504840    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 20:46:09.505113 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505026    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 20:46:09.505359 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505084    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.505359 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505202    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-conf-dir\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.505359 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-systemd-units\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.505359 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-ovnkube-script-lib\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.505359 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptxw\" (UniqueName: \"kubernetes.io/projected/33310520-cb48-4e1e-b89d-f3373d57e646-kube-api-access-rptxw\") pod \"iptables-alerter-h52h5\" (UID: \"33310520-cb48-4e1e-b89d-f3373d57e646\") " pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.505529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505376    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.505529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505414    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-socket-dir-parent\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.505529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:09.505529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-host\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.505529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505521    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhc7q\" (UniqueName: \"kubernetes.io/projected/a404fa55-b2ca-477c-9a5a-23e4e079778a-kube-api-access-zhc7q\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.505719 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-cni-netd\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.505719 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33310520-cb48-4e1e-b89d-f3373d57e646-iptables-alerter-script\") pod \"iptables-alerter-h52h5\" (UID: \"33310520-cb48-4e1e-b89d-f3373d57e646\") " pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.505719 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505591    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.505719 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505617    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-node-log\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.505719 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505645    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-slash\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.505719 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505670    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8576212-b59b-4cf3-9043-7744f159a399-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505718    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4lfs\" (UniqueName: \"kubernetes.io/projected/e8576212-b59b-4cf3-9043-7744f159a399-kube-api-access-r4lfs\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505744    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-sys-fs\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505767    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-var-lib-kubelet\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505809    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505834    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-modprobe-d\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.505883    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-env-overrides\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505934    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-lib-modules\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505964    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-system-cni-dir\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.505991    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfndt\" (UniqueName: \"kubernetes.io/projected/deb437fb-681b-464b-a74b-e670592d6802-kube-api-access-hfndt\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506010    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-run-systemd\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-kubernetes\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506039    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f4938d98-b1e1-4724-97e2-0e948765bd87-hosts-file\") pod \"node-resolver-llqwv\" (UID: \"f4938d98-b1e1-4724-97e2-0e948765bd87\") " pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.506147 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-var-lib-cni-multus\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.507027 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-hostroot\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.507027 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506105    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-run-ovn\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.507027 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-run\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.507027 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506170    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-sys\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.507027 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506193    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-cnibin\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.507027 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-daemon-config\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.507027 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-etc-kubernetes\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.507027 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-etc-openvswitch\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.507027 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.506358    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8576212-b59b-4cf3-9043-7744f159a399-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.508584 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.508058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zb6pl\""
Jul 01 20:46:09.508584 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.508280    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 20:46:09.508584 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.508456    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 20:46:09.519975 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.519952    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 20:46:09.544328 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.544212    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k6dxt"
Jul 01 20:46:09.553341 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.553321    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k6dxt"
Jul 01 20:46:09.590211 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.590168    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" event={"ID":"75bb94aa8471c122d1bbd3ff2e95e5c0","Type":"ContainerStarted","Data":"fa65f54a5e881bc834ab9781175c073ead4673ce5e8c3b120192d381b01b875a"}
Jul 01 20:46:09.591606 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.591580    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" event={"ID":"5835e552044786854dabbf9939043124","Type":"ContainerStarted","Data":"87a3ed0be67da9da1ec00f46a3bafbc42fb460f95cfefdd2d1e3be132e3e8078"}
Jul 01 20:46:09.592515 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.592494    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 20:46:09.607790 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.607759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-device-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.607917 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.607828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mrc\" (UniqueName: \"kubernetes.io/projected/f4938d98-b1e1-4724-97e2-0e948765bd87-kube-api-access-57mrc\") pod \"node-resolver-llqwv\" (UID: \"f4938d98-b1e1-4724-97e2-0e948765bd87\") " pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.608345 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608293    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-device-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-run-netns\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608493    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/634951d8-071a-4335-9c1d-ef274d90307e-host\") pod \"node-ca-ffv5j\" (UID: \"634951d8-071a-4335-9c1d-ef274d90307e\") " pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608498    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-run-netns\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608529    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-sysctl-d\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608561    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-log-socket\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608593    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-ovn-node-metrics-cert\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608668    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-cnibin\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-os-release\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608715    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-sysctl-d\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608737    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-registration-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4938d98-b1e1-4724-97e2-0e948765bd87-tmp-dir\") pod \"node-resolver-llqwv\" (UID: \"f4938d98-b1e1-4724-97e2-0e948765bd87\") " pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608797    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-log-socket\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608801    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-run-multus-certs\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-run-openvswitch\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.609626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-sysconfig\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.608956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a404fa55-b2ca-477c-9a5a-23e4e079778a-tmp\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-kubelet\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-run-netns\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmtgh\" (UniqueName: \"kubernetes.io/projected/634951d8-071a-4335-9c1d-ef274d90307e-kube-api-access-qmtgh\") pod \"node-ca-ffv5j\" (UID: \"634951d8-071a-4335-9c1d-ef274d90307e\") " pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609120    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/beef61b9-6bfd-4bb2-ade2-f535f587aa12-agent-certs\") pod \"konnectivity-agent-n8ht4\" (UID: \"beef61b9-6bfd-4bb2-ade2-f535f587aa12\") " pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-var-lib-kubelet\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-sysctl-conf\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609207    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8576212-b59b-4cf3-9043-7744f159a399-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-conf-dir\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-systemd-units\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.610414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-ovnkube-script-lib\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptxw\" (UniqueName: \"kubernetes.io/projected/33310520-cb48-4e1e-b89d-f3373d57e646-kube-api-access-rptxw\") pod \"iptables-alerter-h52h5\" (UID: \"33310520-cb48-4e1e-b89d-f3373d57e646\") " pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-socket-dir-parent\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609678    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-run-multus-certs\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609815    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-sysconfig\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.609937    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610023    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-kubelet\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610079    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610083    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-socket-dir-parent\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610159    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610204    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-conf-dir\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610260    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-systemd-units\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.611226 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6eb902c9-58f4-4714-b5c4-3259d67349f9-dbus\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610404    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4938d98-b1e1-4724-97e2-0e948765bd87-tmp-dir\") pod \"node-resolver-llqwv\" (UID: \"f4938d98-b1e1-4724-97e2-0e948765bd87\") " pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610419    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-var-lib-kubelet\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610562    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-sysctl-conf\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610650    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-cnibin\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.610895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611011    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-ovnkube-script-lib\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611093    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-run-openvswitch\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611012    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-os-release\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611124    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-host\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhc7q\" (UniqueName: \"kubernetes.io/projected/a404fa55-b2ca-477c-9a5a-23e4e079778a-kube-api-access-zhc7q\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611238    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-host\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-cni-netd\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611331    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-run-netns\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.611946 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33310520-cb48-4e1e-b89d-f3373d57e646-iptables-alerter-script\") pod \"iptables-alerter-h52h5\" (UID: \"33310520-cb48-4e1e-b89d-f3373d57e646\") " pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611370    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-registration-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611500    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/634951d8-071a-4335-9c1d-ef274d90307e-serviceca\") pod \"node-ca-ffv5j\" (UID: \"634951d8-071a-4335-9c1d-ef274d90307e\") " pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611562    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6eb902c9-58f4-4714-b5c4-3259d67349f9-kubelet-config\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-node-log\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-slash\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8576212-b59b-4cf3-9043-7744f159a399-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611727    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8576212-b59b-4cf3-9043-7744f159a399-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611762    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4lfs\" (UniqueName: \"kubernetes.io/projected/e8576212-b59b-4cf3-9043-7744f159a399-kube-api-access-r4lfs\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611803    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-node-log\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611796    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-sys-fs\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611907    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-var-lib-kubelet\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.612641 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.611956    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.611970    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-modprobe-d\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.612064    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs podName:deb437fb-681b-464b-a74b-e670592d6802 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:10.112039767 +0000 UTC m=+3.023374389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs") pod "network-metrics-daemon-sqdk2" (UID: "deb437fb-681b-464b-a74b-e670592d6802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612072    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-modprobe-d\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612184    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/33310520-cb48-4e1e-b89d-f3373d57e646-iptables-alerter-script\") pod \"iptables-alerter-h52h5\" (UID: \"33310520-cb48-4e1e-b89d-f3373d57e646\") " pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612262    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-cni-netd\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612313    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-var-lib-kubelet\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612337    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-sys-fs\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-env-overrides\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612388    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-slash\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612464    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-lib-modules\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612512    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-system-cni-dir\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfndt\" (UniqueName: \"kubernetes.io/projected/deb437fb-681b-464b-a74b-e670592d6802-kube-api-access-hfndt\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-run-systemd\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.613334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612676    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-lib-modules\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612671    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-kubernetes\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612726    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f4938d98-b1e1-4724-97e2-0e948765bd87-hosts-file\") pod \"node-resolver-llqwv\" (UID: \"f4938d98-b1e1-4724-97e2-0e948765bd87\") " pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612740    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8576212-b59b-4cf3-9043-7744f159a399-system-cni-dir\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-var-lib-cni-multus\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612836    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-kubernetes\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612817    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8576212-b59b-4cf3-9043-7744f159a399-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612878    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-env-overrides\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612903    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-run-systemd\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-hostroot\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612967    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f4938d98-b1e1-4724-97e2-0e948765bd87-hosts-file\") pod \"node-resolver-llqwv\" (UID: \"f4938d98-b1e1-4724-97e2-0e948765bd87\") " pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.612999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.613031    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-var-lib-cni-multus\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.613040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-run-ovn\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.613166    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-hostroot\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.614057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.613191    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-run-ovn\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.615385 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615308    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-run\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.615385 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615350    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-sys\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.615530 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-cnibin\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.615530 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-daemon-config\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.615530 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-etc-kubernetes\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.615686 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615628    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-run\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.615742 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615710    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-sys\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.615836 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615779    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-etc-openvswitch\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.615951 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8576212-b59b-4cf3-9043-7744f159a399-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.615951 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615880    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-var-lib-cni-bin\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.615951 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615890    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-etc-kubernetes\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616117 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615922    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/beef61b9-6bfd-4bb2-ade2-f535f587aa12-konnectivity-ca\") pod \"konnectivity-agent-n8ht4\" (UID: \"beef61b9-6bfd-4bb2-ade2-f535f587aa12\") " pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:09.616117 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615993    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-cnibin\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616117 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.615997    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-var-lib-openvswitch\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.616117 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-ovnkube-config\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.616117 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616081    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-socket-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.616377 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-cni-dir\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616377 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616138    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-var-lib-cni-bin\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616377 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-run-k8s-cni-cncf-io\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616377 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616047    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-var-lib-openvswitch\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.616377 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616196    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-daemon-config\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616377 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616231    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-host-run-k8s-cni-cncf-io\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616377 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616313    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-socket-dir\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.616377 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616358    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9vvb\" (UniqueName: \"kubernetes.io/projected/82ea0a51-b205-4ba9-8356-a96baf3f0de6-kube-api-access-z9vvb\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-tuned\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616455    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26q5\" (UniqueName: \"kubernetes.io/projected/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-kube-api-access-g26q5\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616488    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-system-cni-dir\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616521    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-os-release\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82ea0a51-b205-4ba9-8356-a96baf3f0de6-cni-binary-copy\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-systemd\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616617    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-cni-bin\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6zn\" (UniqueName: \"kubernetes.io/projected/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-kube-api-access-4d6zn\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33310520-cb48-4e1e-b89d-f3373d57e646-host-slash\") pod \"iptables-alerter-h52h5\" (UID: \"33310520-cb48-4e1e-b89d-f3373d57e646\") " pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.616780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616776    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-ovnkube-config\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.617354 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616806    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8576212-b59b-4cf3-9043-7744f159a399-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.617354 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616082    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-etc-openvswitch\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.617354 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616811    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/33310520-cb48-4e1e-b89d-f3373d57e646-host-slash\") pod \"iptables-alerter-h52h5\" (UID: \"33310520-cb48-4e1e-b89d-f3373d57e646\") " pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.617354 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616946    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-multus-cni-dir\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.617354 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.616971    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-system-cni-dir\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.617354 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.617047    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.617354 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.617077    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82ea0a51-b205-4ba9-8356-a96baf3f0de6-os-release\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.617354 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.617280    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-host-cni-bin\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.617758 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.617401    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-systemd\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.618881 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.618811    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82ea0a51-b205-4ba9-8356-a96baf3f0de6-cni-binary-copy\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.619682 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.619602    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mrc\" (UniqueName: \"kubernetes.io/projected/f4938d98-b1e1-4724-97e2-0e948765bd87-kube-api-access-57mrc\") pod \"node-resolver-llqwv\" (UID: \"f4938d98-b1e1-4724-97e2-0e948765bd87\") " pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.620385 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.620365    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a404fa55-b2ca-477c-9a5a-23e4e079778a-tmp\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.620544 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.620491    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-ovn-node-metrics-cert\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.623057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.622949    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptxw\" (UniqueName: \"kubernetes.io/projected/33310520-cb48-4e1e-b89d-f3373d57e646-kube-api-access-rptxw\") pod \"iptables-alerter-h52h5\" (UID: \"33310520-cb48-4e1e-b89d-f3373d57e646\") " pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.623651 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.623577    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4lfs\" (UniqueName: \"kubernetes.io/projected/e8576212-b59b-4cf3-9043-7744f159a399-kube-api-access-r4lfs\") pod \"multus-additional-cni-plugins-xb4ww\" (UID: \"e8576212-b59b-4cf3-9043-7744f159a399\") " pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.623783 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.623742    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a404fa55-b2ca-477c-9a5a-23e4e079778a-etc-tuned\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.624237 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.624218    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhc7q\" (UniqueName: \"kubernetes.io/projected/a404fa55-b2ca-477c-9a5a-23e4e079778a-kube-api-access-zhc7q\") pod \"tuned-wjlv6\" (UID: \"a404fa55-b2ca-477c-9a5a-23e4e079778a\") " pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.624905 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.624884    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfndt\" (UniqueName: \"kubernetes.io/projected/deb437fb-681b-464b-a74b-e670592d6802-kube-api-access-hfndt\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:09.626811 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.626763    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26q5\" (UniqueName: \"kubernetes.io/projected/4c9ee537-12c1-4915-a6e4-46fe4784a8c4-kube-api-access-g26q5\") pod \"ovnkube-node-lrvqz\" (UID: \"4c9ee537-12c1-4915-a6e4-46fe4784a8c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.627515 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.627480    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6zn\" (UniqueName: \"kubernetes.io/projected/0fe7e2b8-bb27-4a27-b41a-7aca68de2762-kube-api-access-4d6zn\") pod \"aws-ebs-csi-driver-node-m8rv9\" (UID: \"0fe7e2b8-bb27-4a27-b41a-7aca68de2762\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.628257 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.628183    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9vvb\" (UniqueName: \"kubernetes.io/projected/82ea0a51-b205-4ba9-8356-a96baf3f0de6-kube-api-access-z9vvb\") pod \"multus-85vx9\" (UID: \"82ea0a51-b205-4ba9-8356-a96baf3f0de6\") " pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.717325 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/634951d8-071a-4335-9c1d-ef274d90307e-host\") pod \"node-ca-ffv5j\" (UID: \"634951d8-071a-4335-9c1d-ef274d90307e\") " pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.717325 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717299    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmtgh\" (UniqueName: \"kubernetes.io/projected/634951d8-071a-4335-9c1d-ef274d90307e-kube-api-access-qmtgh\") pod \"node-ca-ffv5j\" (UID: \"634951d8-071a-4335-9c1d-ef274d90307e\") " pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.717325 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717311    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/634951d8-071a-4335-9c1d-ef274d90307e-host\") pod \"node-ca-ffv5j\" (UID: \"634951d8-071a-4335-9c1d-ef274d90307e\") " pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.717583 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/beef61b9-6bfd-4bb2-ade2-f535f587aa12-agent-certs\") pod \"konnectivity-agent-n8ht4\" (UID: \"beef61b9-6bfd-4bb2-ade2-f535f587aa12\") " pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:09.717583 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6eb902c9-58f4-4714-b5c4-3259d67349f9-dbus\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:09.717583 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717431    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:09.717583 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717546    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6eb902c9-58f4-4714-b5c4-3259d67349f9-dbus\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:09.717794 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/634951d8-071a-4335-9c1d-ef274d90307e-serviceca\") pod \"node-ca-ffv5j\" (UID: \"634951d8-071a-4335-9c1d-ef274d90307e\") " pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.717794 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6eb902c9-58f4-4714-b5c4-3259d67349f9-kubelet-config\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:09.717794 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:09.717794 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717716    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/beef61b9-6bfd-4bb2-ade2-f535f587aa12-konnectivity-ca\") pod \"konnectivity-agent-n8ht4\" (UID: \"beef61b9-6bfd-4bb2-ade2-f535f587aa12\") " pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:09.717794 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.717768    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6eb902c9-58f4-4714-b5c4-3259d67349f9-kubelet-config\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:09.717794 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.717784    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:09.718136 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.717895    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret podName:6eb902c9-58f4-4714-b5c4-3259d67349f9 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:10.217873002 +0000 UTC m=+3.129207624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret") pod "global-pull-secret-syncer-zwrwf" (UID: "6eb902c9-58f4-4714-b5c4-3259d67349f9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:09.718275 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.718220    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/634951d8-071a-4335-9c1d-ef274d90307e-serviceca\") pod \"node-ca-ffv5j\" (UID: \"634951d8-071a-4335-9c1d-ef274d90307e\") " pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.718390 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.718365    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/beef61b9-6bfd-4bb2-ade2-f535f587aa12-konnectivity-ca\") pod \"konnectivity-agent-n8ht4\" (UID: \"beef61b9-6bfd-4bb2-ade2-f535f587aa12\") " pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:09.720238 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.720214    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/beef61b9-6bfd-4bb2-ade2-f535f587aa12-agent-certs\") pod \"konnectivity-agent-n8ht4\" (UID: \"beef61b9-6bfd-4bb2-ade2-f535f587aa12\") " pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:09.724434 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.724410    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:09.724544 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.724459    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:09.724544 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.724471    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2874w for pod openshift-network-diagnostics/network-check-target-wzfcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:09.724643 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:09.724573    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w podName:65bb81b4-c408-47ed-a0bb-9798a25206f6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:10.224559403 +0000 UTC m=+3.135894026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2874w" (UniqueName: "kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w") pod "network-check-target-wzfcf" (UID: "65bb81b4-c408-47ed-a0bb-9798a25206f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:09.727023 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.727004    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmtgh\" (UniqueName: \"kubernetes.io/projected/634951d8-071a-4335-9c1d-ef274d90307e-kube-api-access-qmtgh\") pod \"node-ca-ffv5j\" (UID: \"634951d8-071a-4335-9c1d-ef274d90307e\") " pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.800988 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.800959    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xb4ww"
Jul 01 20:46:09.810762 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.810721    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9"
Jul 01 20:46:09.823750 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.823726    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wjlv6"
Jul 01 20:46:09.834906 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.834836    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:09.841880 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.841846    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-llqwv"
Jul 01 20:46:09.850699 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.850679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-85vx9"
Jul 01 20:46:09.858521 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.858498    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h52h5"
Jul 01 20:46:09.866606 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.866582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ffv5j"
Jul 01 20:46:09.873558 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:09.873532    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:10.120823 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:10.120719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:10.121012 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:10.120853    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:10.121012 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:10.120932    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs podName:deb437fb-681b-464b-a74b-e670592d6802 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:11.120916432 +0000 UTC m=+4.032251039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs") pod "network-metrics-daemon-sqdk2" (UID: "deb437fb-681b-464b-a74b-e670592d6802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:10.222140 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:10.222106    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:10.222297 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:10.222248    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:10.222346 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:10.222313    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret podName:6eb902c9-58f4-4714-b5c4-3259d67349f9 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:11.22229679 +0000 UTC m=+4.133631396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret") pod "global-pull-secret-syncer-zwrwf" (UID: "6eb902c9-58f4-4714-b5c4-3259d67349f9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:10.322871 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:10.322817    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:10.323035 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:10.322991    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:10.323035 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:10.323013    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:10.323035 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:10.323027    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2874w for pod openshift-network-diagnostics/network-check-target-wzfcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:10.323198 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:10.323089    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w podName:65bb81b4-c408-47ed-a0bb-9798a25206f6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:11.323068589 +0000 UTC m=+4.234403216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2874w" (UniqueName: "kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w") pod "network-check-target-wzfcf" (UID: "65bb81b4-c408-47ed-a0bb-9798a25206f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:10.555345 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:10.555254    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 20:41:10 +0000 UTC" deadline="2028-02-14 09:43:09.896383415 +0000 UTC"
Jul 01 20:46:10.555345 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:10.555284    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14220h56m59.341103217s"
Jul 01 20:46:10.585514 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:10.585484    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:10.585659 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:10.585587    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:10.821002 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:10.820971    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9ee537_12c1_4915_a6e4_46fe4784a8c4.slice/crio-ed21adcaddc431ed6e04362ecaef1ae34c9818ef9e93804e7c319f6ab864f74d WatchSource:0}: Error finding container ed21adcaddc431ed6e04362ecaef1ae34c9818ef9e93804e7c319f6ab864f74d: Status 404 returned error can't find the container with id ed21adcaddc431ed6e04362ecaef1ae34c9818ef9e93804e7c319f6ab864f74d
Jul 01 20:46:10.821993 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:10.821970    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda404fa55_b2ca_477c_9a5a_23e4e079778a.slice/crio-fa8615de260b84059b058b8640cdb09e29df9688b5f1852afe912baac6518220 WatchSource:0}: Error finding container fa8615de260b84059b058b8640cdb09e29df9688b5f1852afe912baac6518220: Status 404 returned error can't find the container with id fa8615de260b84059b058b8640cdb09e29df9688b5f1852afe912baac6518220
Jul 01 20:46:10.823847 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:10.823787    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeef61b9_6bfd_4bb2_ade2_f535f587aa12.slice/crio-25b6671d290d95c1aa7d10ec608ea3e8a0608e6c0ee8501567ad52d4a37210fe WatchSource:0}: Error finding container 25b6671d290d95c1aa7d10ec608ea3e8a0608e6c0ee8501567ad52d4a37210fe: Status 404 returned error can't find the container with id 25b6671d290d95c1aa7d10ec608ea3e8a0608e6c0ee8501567ad52d4a37210fe
Jul 01 20:46:10.828040 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:10.828021    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ea0a51_b205_4ba9_8356_a96baf3f0de6.slice/crio-fa9faa4599663d12264e5e06a4b5ba3b667cf5cb5ddcea21660041e7ca4c2ab6 WatchSource:0}: Error finding container fa9faa4599663d12264e5e06a4b5ba3b667cf5cb5ddcea21660041e7ca4c2ab6: Status 404 returned error can't find the container with id fa9faa4599663d12264e5e06a4b5ba3b667cf5cb5ddcea21660041e7ca4c2ab6
Jul 01 20:46:10.829074 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:10.829051    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634951d8_071a_4335_9c1d_ef274d90307e.slice/crio-106ac1148b212d8fbf055e240c3406d89fe6d499ed933bcef36ff04c264eb1a6 WatchSource:0}: Error finding container 106ac1148b212d8fbf055e240c3406d89fe6d499ed933bcef36ff04c264eb1a6: Status 404 returned error can't find the container with id 106ac1148b212d8fbf055e240c3406d89fe6d499ed933bcef36ff04c264eb1a6
Jul 01 20:46:10.829750 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:10.829723    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8576212_b59b_4cf3_9043_7744f159a399.slice/crio-2e07eb42954094121c063a96e8833c3368db16093c3ceb6ee2a396dfe03366be WatchSource:0}: Error finding container 2e07eb42954094121c063a96e8833c3368db16093c3ceb6ee2a396dfe03366be: Status 404 returned error can't find the container with id 2e07eb42954094121c063a96e8833c3368db16093c3ceb6ee2a396dfe03366be
Jul 01 20:46:10.831102 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:10.830977    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4938d98_b1e1_4724_97e2_0e948765bd87.slice/crio-85318719b35c36bb5ca4fc1b9497e57fa0ce2337926ea71c0655c4b34f908a28 WatchSource:0}: Error finding container 85318719b35c36bb5ca4fc1b9497e57fa0ce2337926ea71c0655c4b34f908a28: Status 404 returned error can't find the container with id 85318719b35c36bb5ca4fc1b9497e57fa0ce2337926ea71c0655c4b34f908a28
Jul 01 20:46:10.831972 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:10.831685    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33310520_cb48_4e1e_b89d_f3373d57e646.slice/crio-00c881461805734708d0924b71fca9fac74cb4c382058e00e3fd0b618b85dceb WatchSource:0}: Error finding container 00c881461805734708d0924b71fca9fac74cb4c382058e00e3fd0b618b85dceb: Status 404 returned error can't find the container with id 00c881461805734708d0924b71fca9fac74cb4c382058e00e3fd0b618b85dceb
Jul 01 20:46:10.834053 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:10.834019    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe7e2b8_bb27_4a27_b41a_7aca68de2762.slice/crio-407b37823b00b7b1be3458e32cd6b36aafd4b9a5273c450bc591ca8f75e5fff1 WatchSource:0}: Error finding container 407b37823b00b7b1be3458e32cd6b36aafd4b9a5273c450bc591ca8f75e5fff1: Status 404 returned error can't find the container with id 407b37823b00b7b1be3458e32cd6b36aafd4b9a5273c450bc591ca8f75e5fff1
Jul 01 20:46:11.130409 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.130380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:11.130571 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.130553    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:11.130627 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.130617    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs podName:deb437fb-681b-464b-a74b-e670592d6802 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:13.130597796 +0000 UTC m=+6.041932402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs") pod "network-metrics-daemon-sqdk2" (UID: "deb437fb-681b-464b-a74b-e670592d6802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:11.232162 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.232138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:11.232272 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.232237    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:11.232328 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.232285    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret podName:6eb902c9-58f4-4714-b5c4-3259d67349f9 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:13.232273003 +0000 UTC m=+6.143607608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret") pod "global-pull-secret-syncer-zwrwf" (UID: "6eb902c9-58f4-4714-b5c4-3259d67349f9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:11.333450 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.333380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:11.333588 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.333549    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:11.333588 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.333572    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:11.333588 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.333586    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2874w for pod openshift-network-diagnostics/network-check-target-wzfcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:11.333751 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.333653    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w podName:65bb81b4-c408-47ed-a0bb-9798a25206f6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:13.333634193 +0000 UTC m=+6.244968802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2874w" (UniqueName: "kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w") pod "network-check-target-wzfcf" (UID: "65bb81b4-c408-47ed-a0bb-9798a25206f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:11.555908 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.555851    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 20:41:10 +0000 UTC" deadline="2028-02-24 10:51:23.219982123 +0000 UTC"
Jul 01 20:46:11.555908 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.555907    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14462h5m11.664079865s"
Jul 01 20:46:11.585357 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.585287    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:11.585503 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.585440    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:11.585830 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.585674    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:11.585830 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:11.585794    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:11.614028 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.613993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8ht4" event={"ID":"beef61b9-6bfd-4bb2-ade2-f535f587aa12","Type":"ContainerStarted","Data":"25b6671d290d95c1aa7d10ec608ea3e8a0608e6c0ee8501567ad52d4a37210fe"}
Jul 01 20:46:11.617395 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.617271    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerStarted","Data":"ed21adcaddc431ed6e04362ecaef1ae34c9818ef9e93804e7c319f6ab864f74d"}
Jul 01 20:46:11.626664 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.625903    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" event={"ID":"75bb94aa8471c122d1bbd3ff2e95e5c0","Type":"ContainerStarted","Data":"de1f917822f88e61db7b9538541943850da50157fe59120342528e1122834bf2"}
Jul 01 20:46:11.634682 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.634032    2573 generic.go:356] "Generic (PLEG): container finished" podID="5835e552044786854dabbf9939043124" containerID="6329821ed9630c08a3e1c7696b1c0a555805633b34ec26be8a13d6a2080bf137" exitCode=0
Jul 01 20:46:11.634682 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.634106    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" event={"ID":"5835e552044786854dabbf9939043124","Type":"ContainerDied","Data":"6329821ed9630c08a3e1c7696b1c0a555805633b34ec26be8a13d6a2080bf137"}
Jul 01 20:46:11.646145 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.646117    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9" event={"ID":"0fe7e2b8-bb27-4a27-b41a-7aca68de2762","Type":"ContainerStarted","Data":"407b37823b00b7b1be3458e32cd6b36aafd4b9a5273c450bc591ca8f75e5fff1"}
Jul 01 20:46:11.647743 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.646877    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" podStartSLOduration=2.646859221 podStartE2EDuration="2.646859221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:11.646496587 +0000 UTC m=+4.557831219" watchObservedRunningTime="2026-07-01 20:46:11.646859221 +0000 UTC m=+4.558193854"
Jul 01 20:46:11.655651 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.655625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-llqwv" event={"ID":"f4938d98-b1e1-4724-97e2-0e948765bd87","Type":"ContainerStarted","Data":"85318719b35c36bb5ca4fc1b9497e57fa0ce2337926ea71c0655c4b34f908a28"}
Jul 01 20:46:11.657516 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.657443    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ffv5j" event={"ID":"634951d8-071a-4335-9c1d-ef274d90307e","Type":"ContainerStarted","Data":"106ac1148b212d8fbf055e240c3406d89fe6d499ed933bcef36ff04c264eb1a6"}
Jul 01 20:46:11.664619 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.664547    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-85vx9" event={"ID":"82ea0a51-b205-4ba9-8356-a96baf3f0de6","Type":"ContainerStarted","Data":"fa9faa4599663d12264e5e06a4b5ba3b667cf5cb5ddcea21660041e7ca4c2ab6"}
Jul 01 20:46:11.679757 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.679736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wjlv6" event={"ID":"a404fa55-b2ca-477c-9a5a-23e4e079778a","Type":"ContainerStarted","Data":"fa8615de260b84059b058b8640cdb09e29df9688b5f1852afe912baac6518220"}
Jul 01 20:46:11.684057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.683976    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h52h5" event={"ID":"33310520-cb48-4e1e-b89d-f3373d57e646","Type":"ContainerStarted","Data":"00c881461805734708d0924b71fca9fac74cb4c382058e00e3fd0b618b85dceb"}
Jul 01 20:46:11.693190 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:11.693149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb4ww" event={"ID":"e8576212-b59b-4cf3-9043-7744f159a399","Type":"ContainerStarted","Data":"2e07eb42954094121c063a96e8833c3368db16093c3ceb6ee2a396dfe03366be"}
Jul 01 20:46:12.586139 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:12.586102    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:12.586612 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:12.586237    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:12.732708 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:12.732654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" event={"ID":"5835e552044786854dabbf9939043124","Type":"ContainerStarted","Data":"c131df2cd0d3205c87734ef132fbaf4be1ca1911b6463a4b194bd08abe3a2ef6"}
Jul 01 20:46:13.154289 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:13.153614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:13.154289 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.153823    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:13.154289 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.153893    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs podName:deb437fb-681b-464b-a74b-e670592d6802 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:17.153874222 +0000 UTC m=+10.065208832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs") pod "network-metrics-daemon-sqdk2" (UID: "deb437fb-681b-464b-a74b-e670592d6802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:13.255610 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:13.254984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:13.255610 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.255182    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:13.255610 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.255247    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret podName:6eb902c9-58f4-4714-b5c4-3259d67349f9 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:17.255229244 +0000 UTC m=+10.166563855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret") pod "global-pull-secret-syncer-zwrwf" (UID: "6eb902c9-58f4-4714-b5c4-3259d67349f9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:13.356178 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:13.355537    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:13.356178 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.355705    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:13.356178 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.355730    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:13.356178 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.355744    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2874w for pod openshift-network-diagnostics/network-check-target-wzfcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:13.356178 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.355800    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w podName:65bb81b4-c408-47ed-a0bb-9798a25206f6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:17.355781014 +0000 UTC m=+10.267115627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2874w" (UniqueName: "kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w") pod "network-check-target-wzfcf" (UID: "65bb81b4-c408-47ed-a0bb-9798a25206f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:13.587888 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:13.587777    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:13.588352 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.587951    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:13.588352 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:13.588332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:13.588478 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:13.588435    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:14.585534 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:14.585486    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:14.585747 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:14.585634    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:15.587066 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:15.587035    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:15.587683 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:15.587187    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:15.587683 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:15.587555    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:15.587683 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:15.587651    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:16.585649 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:16.585465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:16.585649 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:16.585626    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:17.188170 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:17.188136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:17.188612 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.188277    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:17.188612 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.188345    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs podName:deb437fb-681b-464b-a74b-e670592d6802 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:25.188325106 +0000 UTC m=+18.099659718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs") pod "network-metrics-daemon-sqdk2" (UID: "deb437fb-681b-464b-a74b-e670592d6802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:17.288802 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:17.288766    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:17.288993 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.288943    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:17.289076 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.289014    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret podName:6eb902c9-58f4-4714-b5c4-3259d67349f9 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:25.288991754 +0000 UTC m=+18.200326366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret") pod "global-pull-secret-syncer-zwrwf" (UID: "6eb902c9-58f4-4714-b5c4-3259d67349f9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:17.389122 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:17.389093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:17.389286 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.389251    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:17.389286 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.389273    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:17.389286 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.389286    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2874w for pod openshift-network-diagnostics/network-check-target-wzfcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:17.389441 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.389349    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w podName:65bb81b4-c408-47ed-a0bb-9798a25206f6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:25.38933157 +0000 UTC m=+18.300666179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2874w" (UniqueName: "kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w") pod "network-check-target-wzfcf" (UID: "65bb81b4-c408-47ed-a0bb-9798a25206f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:17.585474 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:17.585356    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:17.585474 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:17.585360    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:17.586911 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.586874    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:17.587039 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:17.586925    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:18.585727 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:18.585687    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:18.586261 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:18.585831    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:19.586830 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:19.586097    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:19.586830 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:19.586233    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:19.586830 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:19.586667    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:19.586830 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:19.586789    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:20.585969 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:20.585938    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:20.586151 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:20.586084    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:21.585203 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:21.585171    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:21.585638 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:21.585171    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:21.585638 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:21.585406    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:21.585638 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:21.585280    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:22.585571 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:22.585540    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:22.585975 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:22.585652    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:23.588084 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:23.588048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:23.588530 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:23.588048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:23.588530 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:23.588169    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:23.588530 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:23.588208    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:24.585995 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:24.585958    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:24.586182 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:24.586066    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:25.256049 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:25.256019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:25.256501 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.256139    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:25.256501 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.256195    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs podName:deb437fb-681b-464b-a74b-e670592d6802 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:41.256178575 +0000 UTC m=+34.167513178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs") pod "network-metrics-daemon-sqdk2" (UID: "deb437fb-681b-464b-a74b-e670592d6802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:25.357138 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:25.357104    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:25.357297 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.357225    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:25.357297 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.357294    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret podName:6eb902c9-58f4-4714-b5c4-3259d67349f9 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:41.357274934 +0000 UTC m=+34.268609541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret") pod "global-pull-secret-syncer-zwrwf" (UID: "6eb902c9-58f4-4714-b5c4-3259d67349f9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:25.458342 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:25.458306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:25.458483 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.458465    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:25.458543 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.458491    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:25.458543 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.458504    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2874w for pod openshift-network-diagnostics/network-check-target-wzfcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:25.458630 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.458558    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w podName:65bb81b4-c408-47ed-a0bb-9798a25206f6 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:41.458544028 +0000 UTC m=+34.369878643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2874w" (UniqueName: "kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w") pod "network-check-target-wzfcf" (UID: "65bb81b4-c408-47ed-a0bb-9798a25206f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:25.585301 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:25.585218    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:25.585457 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.585349    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:25.585457 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:25.585398    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:25.585561 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:25.585492    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:26.585584 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:26.585549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:26.585985 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:26.585656    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:27.588524 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:27.588496    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:27.588951 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:27.588505    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:27.588951 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:27.588584    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:27.588951 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:27.588657    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:28.585970 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:28.585940    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:28.586130 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:28.586050    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:29.585904 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:29.585868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:29.586366 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:29.585875    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:29.586366 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:29.586004    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:29.586366 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:29.586080    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:30.585571 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.585412    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:30.585725 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:30.585643    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:30.772822 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.772769    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9" event={"ID":"0fe7e2b8-bb27-4a27-b41a-7aca68de2762","Type":"ContainerStarted","Data":"4da3ba34b47591cbc080f5199870c8961f7ee544c61d7a2d87e7404f3eb3f83d"}
Jul 01 20:46:30.774370 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.774340    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-llqwv" event={"ID":"f4938d98-b1e1-4724-97e2-0e948765bd87","Type":"ContainerStarted","Data":"be7abdca947105c394eef3fdf328ba4bce0898359c103b2ec7b66a841dda2554"}
Jul 01 20:46:30.775898 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.775874    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ffv5j" event={"ID":"634951d8-071a-4335-9c1d-ef274d90307e","Type":"ContainerStarted","Data":"842b2074a562f2ed3f7f27443902307bb75d86a0587d09e0dc0eb5317dc37dac"}
Jul 01 20:46:30.777426 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.777405    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-85vx9" event={"ID":"82ea0a51-b205-4ba9-8356-a96baf3f0de6","Type":"ContainerStarted","Data":"e239c546aac20f3672fb65f97b6a1a2c62675c3939de53a719353a0f28a84b7a"}
Jul 01 20:46:30.779198 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.779159    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wjlv6" event={"ID":"a404fa55-b2ca-477c-9a5a-23e4e079778a","Type":"ContainerStarted","Data":"d99a75966f4436aacb8a8247080e49fe404eff424b75d4c9f295a90d2c091ea9"}
Jul 01 20:46:30.780727 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.780706    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8576212-b59b-4cf3-9043-7744f159a399" containerID="a05ce351441ce2b97ffe494b49c127c854f9c3b4eaada78b57ca13ab1cec72b4" exitCode=0
Jul 01 20:46:30.780822 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.780774    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb4ww" event={"ID":"e8576212-b59b-4cf3-9043-7744f159a399","Type":"ContainerDied","Data":"a05ce351441ce2b97ffe494b49c127c854f9c3b4eaada78b57ca13ab1cec72b4"}
Jul 01 20:46:30.783053 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.783011    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n8ht4" event={"ID":"beef61b9-6bfd-4bb2-ade2-f535f587aa12","Type":"ContainerStarted","Data":"7f6cb77fc7f64979a4473a606b60c01e6610c31e3d7d14a7850a2587d0c67ca6"}
Jul 01 20:46:30.786622 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.786604    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 20:46:30.787081 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.787057    2573 generic.go:356] "Generic (PLEG): container finished" podID="4c9ee537-12c1-4915-a6e4-46fe4784a8c4" containerID="1c9ed501da2931185af5530153cda23692b47b2f0f35b041a325337922d860a7" exitCode=1
Jul 01 20:46:30.787166 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.787092    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerStarted","Data":"f6939b176007ceadc9984ab08cedf745e7e8455091b8d9640343e73c842ace75"}
Jul 01 20:46:30.787214 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.787175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerStarted","Data":"21612cff0001f03377135d83e7fa812609e5d8ef370bba77918483a6ea552406"}
Jul 01 20:46:30.787267 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.787223    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerStarted","Data":"793cac0dccc39128ed41e6c2f1fbfe499b88792688b23562f9b75f8aa9bcedea"}
Jul 01 20:46:30.787267 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.787250    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerStarted","Data":"261b1231eedc7abd90c7b9b908e5b7c2b07a7d8b2ba11ccaa91963bcd4433b70"}
Jul 01 20:46:30.787394 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.787278    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerDied","Data":"1c9ed501da2931185af5530153cda23692b47b2f0f35b041a325337922d860a7"}
Jul 01 20:46:30.787394 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.787321    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerStarted","Data":"4beee5a6924d06032f1a11cf981b2031dcda98877f4772d48899d541d2e56f42"}
Jul 01 20:46:30.791880 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.791751    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-llqwv" podStartSLOduration=3.47846835 podStartE2EDuration="22.791734899s" totalImagesPullingTime="19.313266549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:30.791279587 +0000 UTC m=+23.702614214" watchObservedRunningTime="2026-07-01 20:46:30.791734899 +0000 UTC m=+23.703069525"
Jul 01 20:46:30.791880 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.791846    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" podStartSLOduration=21.791836546 podStartE2EDuration="21.791836546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:12.750344938 +0000 UTC m=+5.661679570" watchObservedRunningTime="2026-07-01 20:46:30.791836546 +0000 UTC m=+23.703171174"
Jul 01 20:46:30.831041 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.830995    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wjlv6" podStartSLOduration=3.481911977 podStartE2EDuration="22.830982632s" totalImagesPullingTime="19.349070655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:30.8309215 +0000 UTC m=+23.742256127" watchObservedRunningTime="2026-07-01 20:46:30.830982632 +0000 UTC m=+23.742317257"
Jul 01 20:46:30.853362 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.853318    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-85vx9" podStartSLOduration=3.476257716 podStartE2EDuration="22.853304084s" totalImagesPullingTime="19.377046368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:30.852753339 +0000 UTC m=+23.764087965" watchObservedRunningTime="2026-07-01 20:46:30.853304084 +0000 UTC m=+23.764638707"
Jul 01 20:46:30.898434 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.898392    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n8ht4" podStartSLOduration=6.21115577 podStartE2EDuration="21.898379224s" totalImagesPullingTime="15.687223454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:30.898093999 +0000 UTC m=+23.809428626" watchObservedRunningTime="2026-07-01 20:46:30.898379224 +0000 UTC m=+23.809713851"
Jul 01 20:46:30.898580 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:30.898524    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ffv5j" podStartSLOduration=2.5852718660000003 podStartE2EDuration="21.898518148s" totalImagesPullingTime="19.313246282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:30.875699662 +0000 UTC m=+23.787034288" watchObservedRunningTime="2026-07-01 20:46:30.898518148 +0000 UTC m=+23.809852775"
Jul 01 20:46:31.585456 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:31.585430    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:31.585593 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:31.585571    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:31.585676 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:31.585609    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:31.585730 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:31.585684    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:31.596105 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:31.596072    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 20:46:31.791849 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:31.791771    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9" event={"ID":"0fe7e2b8-bb27-4a27-b41a-7aca68de2762","Type":"ContainerStarted","Data":"826dde295c7e3d90f96f607490ae3544ac02569e674cd88d3be3029513a327b2"}
Jul 01 20:46:31.793402 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:31.793371    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h52h5" event={"ID":"33310520-cb48-4e1e-b89d-f3373d57e646","Type":"ContainerStarted","Data":"9d05cb0d730bfa4e3ba80824349e0dbbe27a6c7dcb7e141275ed284398711d9d"}
Jul 01 20:46:31.810172 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:31.810118    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h52h5" podStartSLOduration=4.50040949 podStartE2EDuration="23.810104723s" totalImagesPullingTime="19.309695233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:31.809672895 +0000 UTC m=+24.721007521" watchObservedRunningTime="2026-07-01 20:46:31.810104723 +0000 UTC m=+24.721439349"
Jul 01 20:46:32.565964 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:32.565870    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T20:46:31.596091042Z","UUID":"14b5f165-289a-4e76-b05a-6fc6833ca0dc","Handler":null,"Name":"","Endpoint":""}
Jul 01 20:46:32.567368 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:32.567343    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 20:46:32.567477 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:32.567382    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 20:46:32.585479 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:32.585456    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:32.585600 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:32.585554    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:33.055804 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.055773    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:33.056364 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.056346    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:33.186990 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.186910    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:33.187479 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.187458    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n8ht4"
Jul 01 20:46:33.585810 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.585598    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:33.585964 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.585603    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:33.585964 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:33.585893    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:33.585964 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:33.585929    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:33.801824 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.801798    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 20:46:33.802273 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.802244    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerStarted","Data":"62b0243ef3e5da7b0383eed6de0ea9828f02c1c40459ecf7600553aff07592d0"}
Jul 01 20:46:33.804265 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.804238    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9" event={"ID":"0fe7e2b8-bb27-4a27-b41a-7aca68de2762","Type":"ContainerStarted","Data":"a421e837666a449226054519462268633e91b0d6ead8de4a9ce3d581c20dce85"}
Jul 01 20:46:33.824640 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:33.824598    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8rv9" podStartSLOduration=4.314431429 podStartE2EDuration="25.824583906s" totalImagesPullingTime="21.510152477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:33.823968225 +0000 UTC m=+26.735302856" watchObservedRunningTime="2026-07-01 20:46:33.824583906 +0000 UTC m=+26.735918532"
Jul 01 20:46:34.586034 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:34.586005    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:34.586487 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:34.586116    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:35.585929 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.585703    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:35.586069 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.585698    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:35.586668 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:35.586071    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:35.586668 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:35.585949    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:35.809869 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.809829    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8576212-b59b-4cf3-9043-7744f159a399" containerID="aecbc516c50f8f1634f5d1ad51cca1fad3d27928fb7555126c99a7108289905d" exitCode=0
Jul 01 20:46:35.810010 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.809889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb4ww" event={"ID":"e8576212-b59b-4cf3-9043-7744f159a399","Type":"ContainerDied","Data":"aecbc516c50f8f1634f5d1ad51cca1fad3d27928fb7555126c99a7108289905d"}
Jul 01 20:46:35.813230 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.813214    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 20:46:35.813523 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.813507    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerStarted","Data":"b97a6ca188de7ec1253a64617d9e56331771333a47555bb7a254ce889b029f1c"}
Jul 01 20:46:35.813817 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.813735    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:35.813817 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.813803    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:35.814036 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.813865    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:35.814036 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.813935    2573 scope.go:117] "RemoveContainer" containerID="1c9ed501da2931185af5530153cda23692b47b2f0f35b041a325337922d860a7"
Jul 01 20:46:35.830270 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.830253    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:35.830465 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:35.830449    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:46:36.585564 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:36.585535    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:36.585788 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:36.585639    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:36.820463 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:36.820436    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 20:46:36.820965 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:36.820848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" event={"ID":"4c9ee537-12c1-4915-a6e4-46fe4784a8c4","Type":"ContainerStarted","Data":"b84a483660a881443821d4bb5aa4804defd67803ca6bb403cb8c5078d809cf08"}
Jul 01 20:46:36.853421 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:36.853334    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz" podStartSLOduration=9.452420659 podStartE2EDuration="28.85331314s" totalImagesPullingTime="19.400892481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:36.851472164 +0000 UTC m=+29.762806790" watchObservedRunningTime="2026-07-01 20:46:36.85331314 +0000 UTC m=+29.764647767"
Jul 01 20:46:37.061704 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:37.061676    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sqdk2"]
Jul 01 20:46:37.061843 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:37.061813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:37.062035 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:37.062013    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:37.064289 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:37.064268    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wzfcf"]
Jul 01 20:46:37.064376 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:37.064365    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:37.064470 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:37.064451    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:37.076322 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:37.076299    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zwrwf"]
Jul 01 20:46:37.076395 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:37.076377    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:37.076480 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:37.076459    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:37.824844 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:37.824809    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8576212-b59b-4cf3-9043-7744f159a399" containerID="69e62d8bf15fdab4fbbaed93f6a9ec06ac6a9e7f24529eb032df7bb251f5a96f" exitCode=0
Jul 01 20:46:37.825272 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:37.824881    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb4ww" event={"ID":"e8576212-b59b-4cf3-9043-7744f159a399","Type":"ContainerDied","Data":"69e62d8bf15fdab4fbbaed93f6a9ec06ac6a9e7f24529eb032df7bb251f5a96f"}
Jul 01 20:46:38.586044 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:38.586012    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:38.586205 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:38.586114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:38.586205 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:38.586138    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:38.586205 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:38.586129    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:38.586321 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:38.586217    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:38.586321 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:38.586299    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:39.830606 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:39.830451    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8576212-b59b-4cf3-9043-7744f159a399" containerID="3e17734f3604f2d665ba15369258437bff674e71939bc5ad5a0539b2499e4247" exitCode=0
Jul 01 20:46:39.830984 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:39.830515    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb4ww" event={"ID":"e8576212-b59b-4cf3-9043-7744f159a399","Type":"ContainerDied","Data":"3e17734f3604f2d665ba15369258437bff674e71939bc5ad5a0539b2499e4247"}
Jul 01 20:46:40.585295 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:40.585268    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:40.585449 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:40.585368    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:40.585449 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:40.585378    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:40.585565 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:40.585453    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:40.585565 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:40.585462    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:40.585565 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:40.585546    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:41.277231 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:41.277204    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:41.277688 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:41.277323    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:41.277688 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:41.277378    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs podName:deb437fb-681b-464b-a74b-e670592d6802 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:13.277363658 +0000 UTC m=+66.188698265 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs") pod "network-metrics-daemon-sqdk2" (UID: "deb437fb-681b-464b-a74b-e670592d6802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:41.378180 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:41.378151    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:41.378327 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:41.378308    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:41.378391 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:41.378378    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret podName:6eb902c9-58f4-4714-b5c4-3259d67349f9 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:13.378359324 +0000 UTC m=+66.289693932 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret") pod "global-pull-secret-syncer-zwrwf" (UID: "6eb902c9-58f4-4714-b5c4-3259d67349f9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:41.479051 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:41.479017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:41.479215 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:41.479195    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:41.479256 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:41.479224    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:41.479256 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:41.479234    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2874w for pod openshift-network-diagnostics/network-check-target-wzfcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:41.479319 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:41.479288    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w podName:65bb81b4-c408-47ed-a0bb-9798a25206f6 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:13.479270404 +0000 UTC m=+66.390605027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2874w" (UniqueName: "kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w") pod "network-check-target-wzfcf" (UID: "65bb81b4-c408-47ed-a0bb-9798a25206f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:42.585558 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.585524    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:42.586135 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.585642    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:42.586135 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:42.585647    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzfcf" podUID="65bb81b4-c408-47ed-a0bb-9798a25206f6"
Jul 01 20:46:42.586135 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.585680    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:42.586135 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:42.585771    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqdk2" podUID="deb437fb-681b-464b-a74b-e670592d6802"
Jul 01 20:46:42.586135 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:42.585842    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zwrwf" podUID="6eb902c9-58f4-4714-b5c4-3259d67349f9"
Jul 01 20:46:42.872463 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.872384    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeReady"
Jul 01 20:46:42.872609 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.872529    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 20:46:42.909383 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.909357    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74756dfdf5-52bcr"]
Jul 01 20:46:42.924983 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.924957    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fj468"]
Jul 01 20:46:42.925102 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.925089    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:42.928083 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.928058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 20:46:42.928300 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.928269    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 20:46:42.928701 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.928657    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-txm48\""
Jul 01 20:46:42.928793 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.928749    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 20:46:42.934054 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.933966    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 20:46:42.937682 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.937661    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s6d8g"]
Jul 01 20:46:42.937938 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.937923    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:42.940605 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.940586    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f5twz\""
Jul 01 20:46:42.940706 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.940613    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 20:46:42.940706 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.940652    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 20:46:42.951563 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.951544    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74756dfdf5-52bcr"]
Jul 01 20:46:42.951563 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.951574    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fj468"]
Jul 01 20:46:42.951689 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.951591    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6d8g"]
Jul 01 20:46:42.951689 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.951667    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6d8g"
Jul 01 20:46:42.954456 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.954416    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7ck8j\""
Jul 01 20:46:42.954456 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.954436    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 20:46:42.954626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.954460    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 20:46:42.954626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.954513    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 20:46:42.989190 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989160    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-image-registry-private-configuration\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:42.989283 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k9g7\" (UniqueName: \"kubernetes.io/projected/90dd030d-72b9-4b93-ae9b-24aeec54eca6-kube-api-access-7k9g7\") pod \"ingress-canary-s6d8g\" (UID: \"90dd030d-72b9-4b93-ae9b-24aeec54eca6\") " pod="openshift-ingress-canary/ingress-canary-s6d8g"
Jul 01 20:46:42.989283 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989221    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-trusted-ca\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:42.989283 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-bound-sa-token\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:42.989283 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:42.989454 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989350    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-ca-trust-extracted\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:42.989454 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-installation-pull-secrets\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:42.989454 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989421    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-config-volume\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:42.989454 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:42.989631 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90dd030d-72b9-4b93-ae9b-24aeec54eca6-cert\") pod \"ingress-canary-s6d8g\" (UID: \"90dd030d-72b9-4b93-ae9b-24aeec54eca6\") " pod="openshift-ingress-canary/ingress-canary-s6d8g"
Jul 01 20:46:42.989631 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989503    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-tmp-dir\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:42.989631 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcstf\" (UniqueName: \"kubernetes.io/projected/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-kube-api-access-jcstf\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:42.989631 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989570    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-certificates\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:42.989631 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:42.989621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkpz\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-kube-api-access-bvkpz\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.074728 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.074697    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:43.090815 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.090787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkpz\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-kube-api-access-bvkpz\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.090953 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.090868    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-image-registry-private-configuration\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.090953 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.090901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k9g7\" (UniqueName: \"kubernetes.io/projected/90dd030d-72b9-4b93-ae9b-24aeec54eca6-kube-api-access-7k9g7\") pod \"ingress-canary-s6d8g\" (UID: \"90dd030d-72b9-4b93-ae9b-24aeec54eca6\") " pod="openshift-ingress-canary/ingress-canary-s6d8g"
Jul 01 20:46:43.090953 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.090919    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-trusted-ca\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.090953 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.090941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-bound-sa-token\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.091614 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.091614 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-ca-trust-extracted\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.091614 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-installation-pull-secrets\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.091614 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-config-volume\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:43.091614 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.091315    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:43.091614 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.091329    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74756dfdf5-52bcr: secret "image-registry-tls" not found
Jul 01 20:46:43.091614 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:43.091614 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90dd030d-72b9-4b93-ae9b-24aeec54eca6-cert\") pod \"ingress-canary-s6d8g\" (UID: \"90dd030d-72b9-4b93-ae9b-24aeec54eca6\") " pod="openshift-ingress-canary/ingress-canary-s6d8g"
Jul 01 20:46:43.091614 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.091380    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls podName:08e4dcd0-b1bf-4ec9-a379-68343eec3f23 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:43.591363895 +0000 UTC m=+36.502698513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls") pod "image-registry-74756dfdf5-52bcr" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23") : secret "image-registry-tls" not found
Jul 01 20:46:43.092141 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.091633    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:43.092141 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091630    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-ca-trust-extracted\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.092141 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.091682    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls podName:f8705dc8-bdeb-4766-b015-b5b864ca8ae7 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:43.591665238 +0000 UTC m=+36.503000038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls") pod "dns-default-fj468" (UID: "f8705dc8-bdeb-4766-b015-b5b864ca8ae7") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:43.092141 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-tmp-dir\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:43.092141 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcstf\" (UniqueName: \"kubernetes.io/projected/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-kube-api-access-jcstf\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:43.092141 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-certificates\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.092141 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091942    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-config-volume\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:43.092141 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.091948    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-trusted-ca\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.092472 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.092216    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-tmp-dir\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:43.092472 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.092332    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-certificates\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.095473 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.095454    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90dd030d-72b9-4b93-ae9b-24aeec54eca6-cert\") pod \"ingress-canary-s6d8g\" (UID: \"90dd030d-72b9-4b93-ae9b-24aeec54eca6\") " pod="openshift-ingress-canary/ingress-canary-s6d8g"
Jul 01 20:46:43.095473 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.095466    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-installation-pull-secrets\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.095612 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.095489    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-image-registry-private-configuration\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.100502 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.100462    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k9g7\" (UniqueName: \"kubernetes.io/projected/90dd030d-72b9-4b93-ae9b-24aeec54eca6-kube-api-access-7k9g7\") pod \"ingress-canary-s6d8g\" (UID: \"90dd030d-72b9-4b93-ae9b-24aeec54eca6\") " pod="openshift-ingress-canary/ingress-canary-s6d8g"
Jul 01 20:46:43.100502 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.100478    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-bound-sa-token\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.100842 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.100808    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkpz\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-kube-api-access-bvkpz\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.100842 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.100826    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcstf\" (UniqueName: \"kubernetes.io/projected/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-kube-api-access-jcstf\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:43.175620 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.175526    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:43.208108 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.208080    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-zs75b"]
Jul 01 20:46:43.229711 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.229683    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"]
Jul 01 20:46:43.229883 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.229792    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.232780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.232638    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 01 20:46:43.232780 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.232713    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 20:46:43.232993 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.232782    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 01 20:46:43.232993 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.232918    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tvn4p\""
Jul 01 20:46:43.232993 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.232960    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 20:46:43.238166 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.238147    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 01 20:46:43.241875 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.241845    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"]
Jul 01 20:46:43.241973 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.241944    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.242028 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.241976    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-zs75b"]
Jul 01 20:46:43.244718 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.244696    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8jb8c\""
Jul 01 20:46:43.244817 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.244742    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 20:46:43.244817 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.244777    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 20:46:43.244817 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.244784    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 01 20:46:43.245015 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.244827    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 01 20:46:43.261664 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.261648    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6d8g"
Jul 01 20:46:43.293262 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.293242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97bad7a1-1048-413b-a19c-4768dfca09e5-serving-cert\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.293402 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.293291    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97bad7a1-1048-413b-a19c-4768dfca09e5-tmp\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.293402 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.293317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.293402 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.293355    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.293402 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.293385    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8p7x\" (UniqueName: \"kubernetes.io/projected/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-kube-api-access-f8p7x\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.293602 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.293462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97bad7a1-1048-413b-a19c-4768dfca09e5-snapshots\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.293602 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.293505    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bad7a1-1048-413b-a19c-4768dfca09e5-service-ca-bundle\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.293602 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.293530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78m77\" (UniqueName: \"kubernetes.io/projected/97bad7a1-1048-413b-a19c-4768dfca09e5-kube-api-access-78m77\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.293728 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.293587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bad7a1-1048-413b-a19c-4768dfca09e5-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.315392 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.315368    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-bcd57767c-rwpfk"]
Jul 01 20:46:43.329317 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.329293    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.331378 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.331356    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bcd57767c-rwpfk"]
Jul 01 20:46:43.332307 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.332251    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f88nq\""
Jul 01 20:46:43.332307 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.332296    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 01 20:46:43.332457 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.332446    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 01 20:46:43.332657 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.332638    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 01 20:46:43.332744 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.332669    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 01 20:46:43.332930 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.332904    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 01 20:46:43.333017 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.333007    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 01 20:46:43.394323 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394295    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.394452 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.394452 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8p7x\" (UniqueName: \"kubernetes.io/projected/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-kube-api-access-f8p7x\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.394452 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394378    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97bad7a1-1048-413b-a19c-4768dfca09e5-snapshots\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.394452 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394412    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.394650 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mdz\" (UniqueName: \"kubernetes.io/projected/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-kube-api-access-g4mdz\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.394650 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.394465    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:43.394650 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97bad7a1-1048-413b-a19c-4768dfca09e5-tmp\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.394650 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.394531    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls podName:75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d nodeName:}" failed. No retries permitted until 2026-07-01 20:46:43.89451319 +0000 UTC m=+36.805847798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-6xhzb" (UID: "75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d") : secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:43.394650 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bad7a1-1048-413b-a19c-4768dfca09e5-service-ca-bundle\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.394928 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78m77\" (UniqueName: \"kubernetes.io/projected/97bad7a1-1048-413b-a19c-4768dfca09e5-kube-api-access-78m77\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.394928 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bad7a1-1048-413b-a19c-4768dfca09e5-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.394928 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.394928 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97bad7a1-1048-413b-a19c-4768dfca09e5-serving-cert\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.394928 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394818    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-stats-auth\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.394928 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-default-certificate\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.395207 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.394967    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97bad7a1-1048-413b-a19c-4768dfca09e5-tmp\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.395207 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.395030    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.395207 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.395082    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bad7a1-1048-413b-a19c-4768dfca09e5-service-ca-bundle\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.395207 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.395138    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97bad7a1-1048-413b-a19c-4768dfca09e5-snapshots\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.395685 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.395663    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97bad7a1-1048-413b-a19c-4768dfca09e5-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.397395 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.397372    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97bad7a1-1048-413b-a19c-4768dfca09e5-serving-cert\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.409246 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.409222    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8p7x\" (UniqueName: \"kubernetes.io/projected/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-kube-api-access-f8p7x\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.409246 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.409236    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78m77\" (UniqueName: \"kubernetes.io/projected/97bad7a1-1048-413b-a19c-4768dfca09e5-kube-api-access-78m77\") pod \"insights-operator-5f67b55555-zs75b\" (UID: \"97bad7a1-1048-413b-a19c-4768dfca09e5\") " pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.413514 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.413498    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv"]
Jul 01 20:46:43.439957 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.438171    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"]
Jul 01 20:46:43.440528 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.440088    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv"
Jul 01 20:46:43.443069 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.443047    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:43.443204 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.443182    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q5ds5\""
Jul 01 20:46:43.444757 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.444738    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:43.450264 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.450226    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"]
Jul 01 20:46:43.450341 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.450322    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.453136 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.453117    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:43.453237 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.453137    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 01 20:46:43.453237 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.453164    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-59hcj\""
Jul 01 20:46:43.453495 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.453475    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 01 20:46:43.453593 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.453527    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:43.468390 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.468371    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"]
Jul 01 20:46:43.468499 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.468462    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:43.471149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.471128    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 01 20:46:43.471244 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.471177    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:43.471244 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.471177    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zp92p\""
Jul 01 20:46:43.471361 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.471132    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:43.482831 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.482812    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv"]
Jul 01 20:46:43.482948 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.482845    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"]
Jul 01 20:46:43.482948 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.482875    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"]
Jul 01 20:46:43.483060 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.482944    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.483060 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.482966    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"]
Jul 01 20:46:43.485826 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.485808    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:43.485945 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.485891    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:43.485945 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.485902    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 01 20:46:43.486403 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.486385    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 01 20:46:43.486477 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.486410    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-s9q9w\""
Jul 01 20:46:43.491453 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.491436    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 01 20:46:43.496108 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-default-certificate\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.496189 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496151    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a812588d-2c2a-4559-89f2-c8ade3dec205-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-rfqh7\" (UID: \"a812588d-2c2a-4559-89f2-c8ade3dec205\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.496189 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8r5x\" (UniqueName: \"kubernetes.io/projected/4861b515-af80-46a0-a870-438254fa8c38-kube-api-access-l8r5x\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.496279 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:43.496279 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496233    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4861b515-af80-46a0-a870-438254fa8c38-serving-cert\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.496377 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496285    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7bp\" (UniqueName: \"kubernetes.io/projected/a812588d-2c2a-4559-89f2-c8ade3dec205-kube-api-access-gf7bp\") pod \"service-ca-operator-5d7b7dc95c-rfqh7\" (UID: \"a812588d-2c2a-4559-89f2-c8ade3dec205\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.496434 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496381    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvb4f\" (UniqueName: \"kubernetes.io/projected/d6b4a894-2b09-4c5b-8df9-50f5f5e64e4e-kube-api-access-fvb4f\") pod \"volume-data-source-validator-69d9f679db-gd5rv\" (UID: \"d6b4a894-2b09-4c5b-8df9-50f5f5e64e4e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv"
Jul 01 20:46:43.496434 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.496524 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496442    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vztx\" (UniqueName: \"kubernetes.io/projected/c7f70f9a-122e-4fbf-85e1-9d6db6059227-kube-api-access-8vztx\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:43.496524 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mdz\" (UniqueName: \"kubernetes.io/projected/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-kube-api-access-g4mdz\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.496618 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.496545    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 20:46:43.496618 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.496602    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:43.996579244 +0000 UTC m=+36.907913853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : secret "router-metrics-certs-default" not found
Jul 01 20:46:43.496725 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496644    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a812588d-2c2a-4559-89f2-c8ade3dec205-config\") pod \"service-ca-operator-5d7b7dc95c-rfqh7\" (UID: \"a812588d-2c2a-4559-89f2-c8ade3dec205\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.496725 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496692    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4861b515-af80-46a0-a870-438254fa8c38-config\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.496818 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496727    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4861b515-af80-46a0-a870-438254fa8c38-trusted-ca\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.496818 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.496818 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.496803    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-stats-auth\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.496960 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.496928    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:43.996905842 +0000 UTC m=+36.908240465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : configmap references non-existent config key: service-ca.crt
Jul 01 20:46:43.498889 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.498850    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-default-certificate\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.499062 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.499042    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-stats-auth\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.508754 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.508733    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mdz\" (UniqueName: \"kubernetes.io/projected/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-kube-api-access-g4mdz\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:43.518925 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.518907    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"]
Jul 01 20:46:43.522624 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.522598    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:43.533677 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.533661    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.537160 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.537135    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 01 20:46:43.538792 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.538773    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:43.538900 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.538773    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 01 20:46:43.539365 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.539323    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:43.539585 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.539533    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-x4qrn\""
Jul 01 20:46:43.540767 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.540750    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-zs75b"
Jul 01 20:46:43.550104 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.550084    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"]
Jul 01 20:46:43.597417 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.597397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a812588d-2c2a-4559-89f2-c8ade3dec205-config\") pod \"service-ca-operator-5d7b7dc95c-rfqh7\" (UID: \"a812588d-2c2a-4559-89f2-c8ade3dec205\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.597444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4861b515-af80-46a0-a870-438254fa8c38-config\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.597474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-6j4dv\" (UID: \"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.597504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.597528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4861b515-af80-46a0-a870-438254fa8c38-trusted-ca\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.597575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshmr\" (UniqueName: \"kubernetes.io/projected/e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc-kube-api-access-kshmr\") pod \"kube-storage-version-migrator-operator-79964ff6f9-6j4dv\" (UID: \"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.597607    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.597658    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.597670    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a812588d-2c2a-4559-89f2-c8ade3dec205-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-rfqh7\" (UID: \"a812588d-2c2a-4559-89f2-c8ade3dec205\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.597675    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74756dfdf5-52bcr: secret "image-registry-tls" not found
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.597695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8r5x\" (UniqueName: \"kubernetes.io/projected/4861b515-af80-46a0-a870-438254fa8c38-kube-api-access-l8r5x\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.597770    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.597840    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls podName:f8705dc8-bdeb-4766-b015-b5b864ca8ae7 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:44.597820359 +0000 UTC m=+37.509154979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls") pod "dns-default-fj468" (UID: "f8705dc8-bdeb-4766-b015-b5b864ca8ae7") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:43.598149 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.597871    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls podName:08e4dcd0-b1bf-4ec9-a379-68343eec3f23 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:44.597847657 +0000 UTC m=+37.509182267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls") pod "image-registry-74756dfdf5-52bcr" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23") : secret "image-registry-tls" not found
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.598079    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.598130    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4861b515-af80-46a0-a870-438254fa8c38-serving-cert\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.598161    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a812588d-2c2a-4559-89f2-c8ade3dec205-config\") pod \"service-ca-operator-5d7b7dc95c-rfqh7\" (UID: \"a812588d-2c2a-4559-89f2-c8ade3dec205\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.598162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7bp\" (UniqueName: \"kubernetes.io/projected/a812588d-2c2a-4559-89f2-c8ade3dec205-kube-api-access-gf7bp\") pod \"service-ca-operator-5d7b7dc95c-rfqh7\" (UID: \"a812588d-2c2a-4559-89f2-c8ade3dec205\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.598235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvb4f\" (UniqueName: \"kubernetes.io/projected/d6b4a894-2b09-4c5b-8df9-50f5f5e64e4e-kube-api-access-fvb4f\") pod \"volume-data-source-validator-69d9f679db-gd5rv\" (UID: \"d6b4a894-2b09-4c5b-8df9-50f5f5e64e4e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv"
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.598252    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.598253    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4861b515-af80-46a0-a870-438254fa8c38-config\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.598271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-6j4dv\" (UID: \"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.598296    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls podName:c7f70f9a-122e-4fbf-85e1-9d6db6059227 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:44.098282171 +0000 UTC m=+37.009616778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9lfqm" (UID: "c7f70f9a-122e-4fbf-85e1-9d6db6059227") : secret "samples-operator-tls" not found
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.598346    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vztx\" (UniqueName: \"kubernetes.io/projected/c7f70f9a-122e-4fbf-85e1-9d6db6059227-kube-api-access-8vztx\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:43.598774 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.598419    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4861b515-af80-46a0-a870-438254fa8c38-trusted-ca\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.600431 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.600411    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a812588d-2c2a-4559-89f2-c8ade3dec205-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-rfqh7\" (UID: \"a812588d-2c2a-4559-89f2-c8ade3dec205\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.600567 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.600552    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4861b515-af80-46a0-a870-438254fa8c38-serving-cert\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.607794 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.607771    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7bp\" (UniqueName: \"kubernetes.io/projected/a812588d-2c2a-4559-89f2-c8ade3dec205-kube-api-access-gf7bp\") pod \"service-ca-operator-5d7b7dc95c-rfqh7\" (UID: \"a812588d-2c2a-4559-89f2-c8ade3dec205\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.608061 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.608039    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8r5x\" (UniqueName: \"kubernetes.io/projected/4861b515-af80-46a0-a870-438254fa8c38-kube-api-access-l8r5x\") pod \"console-operator-7fdbdf9c99-r5d2x\" (UID: \"4861b515-af80-46a0-a870-438254fa8c38\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.608156 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.608116    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvb4f\" (UniqueName: \"kubernetes.io/projected/d6b4a894-2b09-4c5b-8df9-50f5f5e64e4e-kube-api-access-fvb4f\") pod \"volume-data-source-validator-69d9f679db-gd5rv\" (UID: \"d6b4a894-2b09-4c5b-8df9-50f5f5e64e4e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv"
Jul 01 20:46:43.608156 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.608114    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vztx\" (UniqueName: \"kubernetes.io/projected/c7f70f9a-122e-4fbf-85e1-9d6db6059227-kube-api-access-8vztx\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:43.622459 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.622434    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:43.699015 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.698935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshmr\" (UniqueName: \"kubernetes.io/projected/e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc-kube-api-access-kshmr\") pod \"kube-storage-version-migrator-operator-79964ff6f9-6j4dv\" (UID: \"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.699174 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.699066    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-6j4dv\" (UID: \"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.699252 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.699231    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-6j4dv\" (UID: \"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.699732 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.699709    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-6j4dv\" (UID: \"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.701555 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.701531    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-6j4dv\" (UID: \"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.712715 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.712697    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshmr\" (UniqueName: \"kubernetes.io/projected/e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc-kube-api-access-kshmr\") pod \"kube-storage-version-migrator-operator-79964ff6f9-6j4dv\" (UID: \"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.752619 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.752587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv"
Jul 01 20:46:43.761763 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.761741    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"
Jul 01 20:46:43.792263 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.792243    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:43.842988 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.842969    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"
Jul 01 20:46:43.901476 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:43.901293    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:43.901476 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.901450    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:43.901750 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:43.901520    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls podName:75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d nodeName:}" failed. No retries permitted until 2026-07-01 20:46:44.901500094 +0000 UTC m=+37.812834711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-6xhzb" (UID: "75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d") : secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:44.006403 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.005251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:44.006403 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.005598    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:44.006403 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.005828    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:45.005810054 +0000 UTC m=+37.917144677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : configmap references non-existent config key: service-ca.crt
Jul 01 20:46:44.006403 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.006305    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 20:46:44.006403 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.006360    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:45.006344147 +0000 UTC m=+37.917678754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : secret "router-metrics-certs-default" not found
Jul 01 20:46:44.098414 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.098374    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6d8g"]
Jul 01 20:46:44.101543 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.101519    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"]
Jul 01 20:46:44.102257 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:44.102219    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90dd030d_72b9_4b93_ae9b_24aeec54eca6.slice/crio-cbb9a079ca671e0cbc8f39785e798a24172d7fb4187708cf77dc1ea92abc2268 WatchSource:0}: Error finding container cbb9a079ca671e0cbc8f39785e798a24172d7fb4187708cf77dc1ea92abc2268: Status 404 returned error can't find the container with id cbb9a079ca671e0cbc8f39785e798a24172d7fb4187708cf77dc1ea92abc2268
Jul 01 20:46:44.105311 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:44.105272    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4861b515_af80_46a0_a870_438254fa8c38.slice/crio-734d535d74396e0cf5b1b925803e42755fac87c80df9fb463ffdd777acf7a335 WatchSource:0}: Error finding container 734d535d74396e0cf5b1b925803e42755fac87c80df9fb463ffdd777acf7a335: Status 404 returned error can't find the container with id 734d535d74396e0cf5b1b925803e42755fac87c80df9fb463ffdd777acf7a335
Jul 01 20:46:44.105555 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.105499    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv"]
Jul 01 20:46:44.107344 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.106210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:44.107344 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.106423    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 20:46:44.107344 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.106487    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls podName:c7f70f9a-122e-4fbf-85e1-9d6db6059227 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:45.106469689 +0000 UTC m=+38.017804307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9lfqm" (UID: "c7f70f9a-122e-4fbf-85e1-9d6db6059227") : secret "samples-operator-tls" not found
Jul 01 20:46:44.110146 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:44.110122    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fc71dc_1f85_422e_b6a9_2cd2c4fe28dc.slice/crio-73313a79059eaae68c4d91121afddfbf8be81ddab0b42a80223fd4957afe811a WatchSource:0}: Error finding container 73313a79059eaae68c4d91121afddfbf8be81ddab0b42a80223fd4957afe811a: Status 404 returned error can't find the container with id 73313a79059eaae68c4d91121afddfbf8be81ddab0b42a80223fd4957afe811a
Jul 01 20:46:44.118876 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.118836    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-zs75b"]
Jul 01 20:46:44.121335 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.121307    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv"]
Jul 01 20:46:44.124022 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:44.123983    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b4a894_2b09_4c5b_8df9_50f5f5e64e4e.slice/crio-4a2fcbdbbc58b921daa5ed7daa3b7645129a105f32aefd9ec29029df793fb5c0 WatchSource:0}: Error finding container 4a2fcbdbbc58b921daa5ed7daa3b7645129a105f32aefd9ec29029df793fb5c0: Status 404 returned error can't find the container with id 4a2fcbdbbc58b921daa5ed7daa3b7645129a105f32aefd9ec29029df793fb5c0
Jul 01 20:46:44.125870 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:44.125831    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bad7a1_1048_413b_a19c_4768dfca09e5.slice/crio-e9da250fa5a6b91ac9eae45a3271a5154e3dcd8673ad2faf8fe6aa5afb6e9762 WatchSource:0}: Error finding container e9da250fa5a6b91ac9eae45a3271a5154e3dcd8673ad2faf8fe6aa5afb6e9762: Status 404 returned error can't find the container with id e9da250fa5a6b91ac9eae45a3271a5154e3dcd8673ad2faf8fe6aa5afb6e9762
Jul 01 20:46:44.132231 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.132212    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7"]
Jul 01 20:46:44.135429 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:44.135409    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda812588d_2c2a_4559_89f2_c8ade3dec205.slice/crio-aec32716326e894d7ededabc0fd4bb1ce95342d6b241212bbd42805a8c7ca457 WatchSource:0}: Error finding container aec32716326e894d7ededabc0fd4bb1ce95342d6b241212bbd42805a8c7ca457: Status 404 returned error can't find the container with id aec32716326e894d7ededabc0fd4bb1ce95342d6b241212bbd42805a8c7ca457
Jul 01 20:46:44.585819 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.585790    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:46:44.586075 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.585790    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:46:44.586075 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.585790    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:46:44.588814 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.588794    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 20:46:44.588952 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.588936    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 20:46:44.589029 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.589015    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v6s46\""
Jul 01 20:46:44.589029 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.589022    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 20:46:44.589093 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.589016    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 20:46:44.591089 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.591064    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4g6n6\""
Jul 01 20:46:44.609993 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.609968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:44.610289 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.610012    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:44.610289 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.610104    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:44.610289 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.610118    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74756dfdf5-52bcr: secret "image-registry-tls" not found
Jul 01 20:46:44.610289 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.610126    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:44.610289 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.610165    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls podName:08e4dcd0-b1bf-4ec9-a379-68343eec3f23 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:46.610148938 +0000 UTC m=+39.521483542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls") pod "image-registry-74756dfdf5-52bcr" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23") : secret "image-registry-tls" not found
Jul 01 20:46:44.610289 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.610179    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls podName:f8705dc8-bdeb-4766-b015-b5b864ca8ae7 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:46.610172865 +0000 UTC m=+39.521507469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls") pod "dns-default-fj468" (UID: "f8705dc8-bdeb-4766-b015-b5b864ca8ae7") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:44.843763 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.843695    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7" event={"ID":"a812588d-2c2a-4559-89f2-c8ade3dec205","Type":"ContainerStarted","Data":"aec32716326e894d7ededabc0fd4bb1ce95342d6b241212bbd42805a8c7ca457"}
Jul 01 20:46:44.846121 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.846090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-zs75b" event={"ID":"97bad7a1-1048-413b-a19c-4768dfca09e5","Type":"ContainerStarted","Data":"e9da250fa5a6b91ac9eae45a3271a5154e3dcd8673ad2faf8fe6aa5afb6e9762"}
Jul 01 20:46:44.851255 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.851229    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8576212-b59b-4cf3-9043-7744f159a399" containerID="215ac0666b18abc33b29b87767c38cdc1a933c9c328509b89f46ba61cfcaefa9" exitCode=0
Jul 01 20:46:44.851339 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.851301    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb4ww" event={"ID":"e8576212-b59b-4cf3-9043-7744f159a399","Type":"ContainerDied","Data":"215ac0666b18abc33b29b87767c38cdc1a933c9c328509b89f46ba61cfcaefa9"}
Jul 01 20:46:44.860240 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.860217    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv" event={"ID":"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc","Type":"ContainerStarted","Data":"73313a79059eaae68c4d91121afddfbf8be81ddab0b42a80223fd4957afe811a"}
Jul 01 20:46:44.862333 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.862277    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6d8g" event={"ID":"90dd030d-72b9-4b93-ae9b-24aeec54eca6","Type":"ContainerStarted","Data":"cbb9a079ca671e0cbc8f39785e798a24172d7fb4187708cf77dc1ea92abc2268"}
Jul 01 20:46:44.864984 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.864925    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv" event={"ID":"d6b4a894-2b09-4c5b-8df9-50f5f5e64e4e","Type":"ContainerStarted","Data":"4a2fcbdbbc58b921daa5ed7daa3b7645129a105f32aefd9ec29029df793fb5c0"}
Jul 01 20:46:44.866600 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.866556    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" event={"ID":"4861b515-af80-46a0-a870-438254fa8c38","Type":"ContainerStarted","Data":"734d535d74396e0cf5b1b925803e42755fac87c80df9fb463ffdd777acf7a335"}
Jul 01 20:46:44.912519 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:44.911872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:44.912519 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.912070    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:44.912519 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:44.912151    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls podName:75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d nodeName:}" failed. No retries permitted until 2026-07-01 20:46:46.912130136 +0000 UTC m=+39.823464755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-6xhzb" (UID: "75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d") : secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:45.014912 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.014876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:45.015028 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:45.014971    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 20:46:45.015405 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.015029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:45.015405 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:45.015044    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:47.015024328 +0000 UTC m=+39.926358950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : secret "router-metrics-certs-default" not found
Jul 01 20:46:45.015405 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:45.015126    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:47.015110409 +0000 UTC m=+39.926445028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : configmap references non-existent config key: service-ca.crt
Jul 01 20:46:45.071501 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.071411    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:45.119316 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.118092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:45.119316 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:45.118494    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 20:46:45.119316 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:45.118569    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls podName:c7f70f9a-122e-4fbf-85e1-9d6db6059227 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:47.118550561 +0000 UTC m=+40.029885178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9lfqm" (UID: "c7f70f9a-122e-4fbf-85e1-9d6db6059227") : secret "samples-operator-tls" not found
Jul 01 20:46:45.150090 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.150041    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:45.246607 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.246485    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:45.609409 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.608310    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-86srq"]
Jul 01 20:46:45.611253 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.610734    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-86srq"
Jul 01 20:46:45.614097 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.613894    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rcbmn\""
Jul 01 20:46:45.625230 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.625156    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-86srq"]
Jul 01 20:46:45.724089 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.723958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx56k\" (UniqueName: \"kubernetes.io/projected/dae54076-5a60-4d41-9b05-c281a0c1de43-kube-api-access-lx56k\") pod \"network-check-source-56499946bc-86srq\" (UID: \"dae54076-5a60-4d41-9b05-c281a0c1de43\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-86srq"
Jul 01 20:46:45.824900 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.824573    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx56k\" (UniqueName: \"kubernetes.io/projected/dae54076-5a60-4d41-9b05-c281a0c1de43-kube-api-access-lx56k\") pod \"network-check-source-56499946bc-86srq\" (UID: \"dae54076-5a60-4d41-9b05-c281a0c1de43\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-86srq"
Jul 01 20:46:45.846081 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.845886    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx56k\" (UniqueName: \"kubernetes.io/projected/dae54076-5a60-4d41-9b05-c281a0c1de43-kube-api-access-lx56k\") pod \"network-check-source-56499946bc-86srq\" (UID: \"dae54076-5a60-4d41-9b05-c281a0c1de43\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-86srq"
Jul 01 20:46:45.884328 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.883257    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8576212-b59b-4cf3-9043-7744f159a399" containerID="0e4aab5d4937e0b793fda6c87541f4c8c6646b69444d489891df0fc7ef2e5b23" exitCode=0
Jul 01 20:46:45.884328 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.883304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb4ww" event={"ID":"e8576212-b59b-4cf3-9043-7744f159a399","Type":"ContainerDied","Data":"0e4aab5d4937e0b793fda6c87541f4c8c6646b69444d489891df0fc7ef2e5b23"}
Jul 01 20:46:45.925894 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:45.925659    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-86srq"
Jul 01 20:46:46.117660 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.117627    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:46.143718 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.142635    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-86srq"]
Jul 01 20:46:46.232537 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.232396    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:46.634984 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.634777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:46.635745 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.635029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:46.635745 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:46.635266    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:46.635745 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:46.635321    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls podName:f8705dc8-bdeb-4766-b015-b5b864ca8ae7 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:50.635308292 +0000 UTC m=+43.546642896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls") pod "dns-default-fj468" (UID: "f8705dc8-bdeb-4766-b015-b5b864ca8ae7") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:46.635745 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:46.635374    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:46.635745 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:46.635384    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74756dfdf5-52bcr: secret "image-registry-tls" not found
Jul 01 20:46:46.635745 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:46.635416    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls podName:08e4dcd0-b1bf-4ec9-a379-68343eec3f23 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:50.635404838 +0000 UTC m=+43.546739442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls") pod "image-registry-74756dfdf5-52bcr" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23") : secret "image-registry-tls" not found
Jul 01 20:46:46.700200 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.700145    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:46.802217 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.802185    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:46.890179 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.890096    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb4ww" event={"ID":"e8576212-b59b-4cf3-9043-7744f159a399","Type":"ContainerStarted","Data":"c5d37335aae20ce363f96ef32d152ef383b7c18c586d053d07c9ffcaa207b7a0"}
Jul 01 20:46:46.892017 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.891988    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv" event={"ID":"d6b4a894-2b09-4c5b-8df9-50f5f5e64e4e","Type":"ContainerStarted","Data":"25eb198871d1c973b7c3a427e1baa65598af7c66f5e42d55a573d6e314bec6f1"}
Jul 01 20:46:46.893128 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.893107    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-86srq" event={"ID":"dae54076-5a60-4d41-9b05-c281a0c1de43","Type":"ContainerStarted","Data":"ad786b66a97a38507c13ef5a1747595cfdfb2e4ec1525668c93a3c9f1fdf26a7"}
Jul 01 20:46:46.915728 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.915679    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xb4ww" podStartSLOduration=8.723811858 podStartE2EDuration="38.91565882s" totalImagesPullingTime="30.191846962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:46.913428029 +0000 UTC m=+39.824762656" watchObservedRunningTime="2026-07-01 20:46:46.91565882 +0000 UTC m=+39.826993449"
Jul 01 20:46:46.929137 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.929098    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-gd5rv" podStartSLOduration=2.2519734 podStartE2EDuration="3.92908162s" totalImagesPullingTime="1.67710822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:46.928233638 +0000 UTC m=+39.839568264" watchObservedRunningTime="2026-07-01 20:46:46.92908162 +0000 UTC m=+39.840416244"
Jul 01 20:46:46.938049 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:46.937907    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:46.938165 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:46.938096    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:46.938165 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:46.938152    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls podName:75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d nodeName:}" failed. No retries permitted until 2026-07-01 20:46:50.938135466 +0000 UTC m=+43.849470071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-6xhzb" (UID: "75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d") : secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:47.039243 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:47.039207    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:47.039388 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:47.039333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:47.039388 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:47.039347    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 20:46:47.039497 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:47.039414    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:51.039393669 +0000 UTC m=+43.950728288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : secret "router-metrics-certs-default" not found
Jul 01 20:46:47.039497 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:47.039493    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:51.039478477 +0000 UTC m=+43.950813081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : configmap references non-existent config key: service-ca.crt
Jul 01 20:46:47.050312 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:47.050288    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:47.140365 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:47.140269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:47.140491 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:47.140405    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 20:46:47.140491 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:47.140468    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls podName:c7f70f9a-122e-4fbf-85e1-9d6db6059227 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:51.140454685 +0000 UTC m=+44.051789296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9lfqm" (UID: "c7f70f9a-122e-4fbf-85e1-9d6db6059227") : secret "samples-operator-tls" not found
Jul 01 20:46:47.150922 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:47.150892    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:47.631039 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:47.631005    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:47.685771 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:47.685742    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:49.038194 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:49.038133    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:50.610269 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.610235    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"]
Jul 01 20:46:50.634075 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.634051    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"]
Jul 01 20:46:50.634217 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.634159    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:50.637495 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.637476    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2bnfz\""
Jul 01 20:46:50.637626 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.637476    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 01 20:46:50.639460 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.639435    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 01 20:46:50.673103 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.673078    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:50.673238 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.673119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:50.673238 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.673213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:50.673350 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:50.673238    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:50.673350 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:50.673250    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74756dfdf5-52bcr: secret "image-registry-tls" not found
Jul 01 20:46:50.673350 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.673251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:50.673350 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:50.673293    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls podName:08e4dcd0-b1bf-4ec9-a379-68343eec3f23 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:58.673277323 +0000 UTC m=+51.584611927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls") pod "image-registry-74756dfdf5-52bcr" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23") : secret "image-registry-tls" not found
Jul 01 20:46:50.673594 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:50.673356    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:50.673594 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:50.673407    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls podName:f8705dc8-bdeb-4766-b015-b5b864ca8ae7 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:58.673392572 +0000 UTC m=+51.584727201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls") pod "dns-default-fj468" (UID: "f8705dc8-bdeb-4766-b015-b5b864ca8ae7") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:50.736703 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.736663    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:50.773822 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.773787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:50.773949 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.773886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:50.774037 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:50.774024    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:46:50.774141 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:50.774087    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert podName:3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:51.274070666 +0000 UTC m=+44.185405273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-fhkqt" (UID: "3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0") : secret "networking-console-plugin-cert" not found
Jul 01 20:46:50.774567 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.774548    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:50.837415 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.837366    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:50.936764 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.936663    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:50.975215 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:50.975186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:50.975354 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:50.975336    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:50.975422 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:50.975395    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls podName:75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d nodeName:}" failed. No retries permitted until 2026-07-01 20:46:58.975379026 +0000 UTC m=+51.886713630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-6xhzb" (UID: "75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d") : secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:51.038882 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.038837    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:51.076279 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.076251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:51.076465 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.076384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:51.076530 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:51.076502    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:59.076482003 +0000 UTC m=+51.987816619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : configmap references non-existent config key: service-ca.crt
Jul 01 20:46:51.076591 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:51.076530    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 20:46:51.076591 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:51.076586    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:46:59.076569474 +0000 UTC m=+51.987904079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : secret "router-metrics-certs-default" not found
Jul 01 20:46:51.177718 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.177691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:51.177900 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:51.177884    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 20:46:51.177984 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:51.177972    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls podName:c7f70f9a-122e-4fbf-85e1-9d6db6059227 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:59.177952841 +0000 UTC m=+52.089287458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9lfqm" (UID: "c7f70f9a-122e-4fbf-85e1-9d6db6059227") : secret "samples-operator-tls" not found
Jul 01 20:46:51.279203 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.279135    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:51.279362 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:51.279287    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:46:51.279424 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:51.279362    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert podName:3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:52.279346132 +0000 UTC m=+45.190680740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-fhkqt" (UID: "3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0") : secret "networking-console-plugin-cert" not found
Jul 01 20:46:51.910558 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.910497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7" event={"ID":"a812588d-2c2a-4559-89f2-c8ade3dec205","Type":"ContainerStarted","Data":"b6c7177d89c8ffee2df145ec01cd7f66942990b176536d0dff7675d1435f6df2"}
Jul 01 20:46:51.912464 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.912413    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-zs75b" event={"ID":"97bad7a1-1048-413b-a19c-4768dfca09e5","Type":"ContainerStarted","Data":"a22c0e7232b2ff011910b2ab1a60b827358a504631e588c931e49ff6b8a47546"}
Jul 01 20:46:51.914186 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.914159    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv" event={"ID":"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc","Type":"ContainerStarted","Data":"57a4dd4a1bc7ddd15b4669f573e46121535f11ea59bb5cb0b4728e7325ad0e7c"}
Jul 01 20:46:51.915968 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.915940    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6d8g" event={"ID":"90dd030d-72b9-4b93-ae9b-24aeec54eca6","Type":"ContainerStarted","Data":"eacde2321a4e81d58005d7bcad016a0996e5fec8dd2a2e29777592f840431eef"}
Jul 01 20:46:51.917664 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.917646    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/0.log"
Jul 01 20:46:51.917762 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.917684    2573 generic.go:356] "Generic (PLEG): container finished" podID="4861b515-af80-46a0-a870-438254fa8c38" containerID="f74f8ea982c83d2b9d2a3df7f879a2cd5c6f07e54657c6950bd9fa0ea9363c0e" exitCode=255
Jul 01 20:46:51.917762 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.917713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" event={"ID":"4861b515-af80-46a0-a870-438254fa8c38","Type":"ContainerDied","Data":"f74f8ea982c83d2b9d2a3df7f879a2cd5c6f07e54657c6950bd9fa0ea9363c0e"}
Jul 01 20:46:51.917992 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.917976    2573 scope.go:117] "RemoveContainer" containerID="f74f8ea982c83d2b9d2a3df7f879a2cd5c6f07e54657c6950bd9fa0ea9363c0e"
Jul 01 20:46:51.929073 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.929016    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7" podStartSLOduration=1.629812454 podStartE2EDuration="8.92899825s" totalImagesPullingTime="7.299185796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:51.92767757 +0000 UTC m=+44.839012203" watchObservedRunningTime="2026-07-01 20:46:51.92899825 +0000 UTC m=+44.840332878"
Jul 01 20:46:51.949323 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:51.949190    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv" podStartSLOduration=1.625225549 podStartE2EDuration="8.949173454s" totalImagesPullingTime="7.323947905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:51.947943203 +0000 UTC m=+44.859277830" watchObservedRunningTime="2026-07-01 20:46:51.949173454 +0000 UTC m=+44.860508081"
Jul 01 20:46:52.014525 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.014471    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s6d8g" podStartSLOduration=2.685313666 podStartE2EDuration="10.014448023s" totalImagesPullingTime="7.329134357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:51.990690217 +0000 UTC m=+44.902024845" watchObservedRunningTime="2026-07-01 20:46:52.014448023 +0000 UTC m=+44.925782650"
Jul 01 20:46:52.289002 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.288931    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:52.289141 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:52.289123    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:46:52.289198 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:52.289184    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert podName:3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:54.289166609 +0000 UTC m=+47.200501220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-fhkqt" (UID: "3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0") : secret "networking-console-plugin-cert" not found
Jul 01 20:46:52.766957 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.766925    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:52.803904 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.803822    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-zs75b" podStartSLOduration=2.495463258 podStartE2EDuration="9.803804797s" totalImagesPullingTime="7.308341539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:52.014768131 +0000 UTC m=+44.926102753" watchObservedRunningTime="2026-07-01 20:46:52.803804797 +0000 UTC m=+45.715139437"
Jul 01 20:46:52.804811 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.804794    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb"]
Jul 01 20:46:52.824111 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.824048    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb"]
Jul 01 20:46:52.824269 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.824165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb"
Jul 01 20:46:52.829103 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.828993    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 01 20:46:52.829741 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.829716    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-szdqv\""
Jul 01 20:46:52.829992 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.829976    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:52.872106 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.872082    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:52.894199 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.894173    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6zm\" (UniqueName: \"kubernetes.io/projected/b484c66a-201d-48bd-9925-25bc2c2170af-kube-api-access-gb6zm\") pod \"migrator-786f9fbc5-2hghb\" (UID: \"b484c66a-201d-48bd-9925-25bc2c2170af\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb"
Jul 01 20:46:52.927088 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.927060    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-86srq" event={"ID":"dae54076-5a60-4d41-9b05-c281a0c1de43","Type":"ContainerStarted","Data":"145a79168c661f712de8aaa0baeba064ce6003214e4f38f74c33c33c66d7c240"}
Jul 01 20:46:52.928847 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.928831    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/1.log"
Jul 01 20:46:52.929270 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.929254    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/0.log"
Jul 01 20:46:52.929344 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.929289    2573 generic.go:356] "Generic (PLEG): container finished" podID="4861b515-af80-46a0-a870-438254fa8c38" containerID="372189b8d38d3890b737a6c8d2e89cdb08bc70e373cd44446bc0b038ee2c733b" exitCode=255
Jul 01 20:46:52.929403 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.929354    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" event={"ID":"4861b515-af80-46a0-a870-438254fa8c38","Type":"ContainerDied","Data":"372189b8d38d3890b737a6c8d2e89cdb08bc70e373cd44446bc0b038ee2c733b"}
Jul 01 20:46:52.929455 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.929396    2573 scope.go:117] "RemoveContainer" containerID="f74f8ea982c83d2b9d2a3df7f879a2cd5c6f07e54657c6950bd9fa0ea9363c0e"
Jul 01 20:46:52.929609 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.929589    2573 scope.go:117] "RemoveContainer" containerID="372189b8d38d3890b737a6c8d2e89cdb08bc70e373cd44446bc0b038ee2c733b"
Jul 01 20:46:52.929781 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:52.929750    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-r5d2x_openshift-console-operator(4861b515-af80-46a0-a870-438254fa8c38)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" podUID="4861b515-af80-46a0-a870-438254fa8c38"
Jul 01 20:46:52.931645 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.931624    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:52.945068 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.945031    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56499946bc-86srq" podStartSLOduration=2.221496582 podStartE2EDuration="7.945019759s" totalImagesPullingTime="5.723523177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:52.943947 +0000 UTC m=+45.855281628" watchObservedRunningTime="2026-07-01 20:46:52.945019759 +0000 UTC m=+45.856354423"
Jul 01 20:46:52.995925 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.995898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6zm\" (UniqueName: \"kubernetes.io/projected/b484c66a-201d-48bd-9925-25bc2c2170af-kube-api-access-gb6zm\") pod \"migrator-786f9fbc5-2hghb\" (UID: \"b484c66a-201d-48bd-9925-25bc2c2170af\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb"
Jul 01 20:46:52.998529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:52.998503    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:53.005804 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.005785    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6zm\" (UniqueName: \"kubernetes.io/projected/b484c66a-201d-48bd-9925-25bc2c2170af-kube-api-access-gb6zm\") pod \"migrator-786f9fbc5-2hghb\" (UID: \"b484c66a-201d-48bd-9925-25bc2c2170af\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb"
Jul 01 20:46:53.130807 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.130774    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sk28n"]
Jul 01 20:46:53.135900 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.135874    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb"
Jul 01 20:46:53.158042 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.158020    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sk28n"]
Jul 01 20:46:53.158170 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.158116    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.161440 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.161416    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 20:46:53.161547 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.161418    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 20:46:53.161727 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.161712    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9fsnr\""
Jul 01 20:46:53.185028 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.185002    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:53.252788 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.252734    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:53.292985 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.292869    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb"]
Jul 01 20:46:53.295574 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:53.295550    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb484c66a_201d_48bd_9925_25bc2c2170af.slice/crio-bcda6be965475179df335c1a9b2bc1b689883e0bab84a93d4481b17fd32038f5 WatchSource:0}: Error finding container bcda6be965475179df335c1a9b2bc1b689883e0bab84a93d4481b17fd32038f5: Status 404 returned error can't find the container with id bcda6be965475179df335c1a9b2bc1b689883e0bab84a93d4481b17fd32038f5
Jul 01 20:46:53.298637 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.298620    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4ad4488-3e80-4ce5-84c3-1b6524053abe-crio-socket\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.298724 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.298710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ad4488-3e80-4ce5-84c3-1b6524053abe-data-volume\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.298777 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.298745    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.298777 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.298765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.298886 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.298798    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgtcx\" (UniqueName: \"kubernetes.io/projected/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-api-access-xgtcx\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.399685 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.399605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgtcx\" (UniqueName: \"kubernetes.io/projected/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-api-access-xgtcx\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.399832 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.399724    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4ad4488-3e80-4ce5-84c3-1b6524053abe-crio-socket\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.399927 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.399830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ad4488-3e80-4ce5-84c3-1b6524053abe-data-volume\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.399927 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.399890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.400032 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.399927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.400032 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.399978    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4ad4488-3e80-4ce5-84c3-1b6524053abe-crio-socket\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.400032 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:53.400010    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:46:53.400032 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:53.400024    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 20:46:53.400177 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:53.400096    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls podName:f4ad4488-3e80-4ce5-84c3-1b6524053abe nodeName:}" failed. No retries permitted until 2026-07-01 20:46:53.900061304 +0000 UTC m=+46.811395934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sk28n" (UID: "f4ad4488-3e80-4ce5-84c3-1b6524053abe") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:46:53.400177 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:53.400116    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm podName:f4ad4488-3e80-4ce5-84c3-1b6524053abe nodeName:}" failed. No retries permitted until 2026-07-01 20:46:53.900105836 +0000 UTC m=+46.811440440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sk28n" (UID: "f4ad4488-3e80-4ce5-84c3-1b6524053abe") : configmap "kube-rbac-proxy" not found
Jul 01 20:46:53.400256 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.400194    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ad4488-3e80-4ce5-84c3-1b6524053abe-data-volume\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.408606 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.408588    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgtcx\" (UniqueName: \"kubernetes.io/projected/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-api-access-xgtcx\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.434186 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.434162    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:53.709536 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.709459    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:53.792989 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.792961    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:53.793106 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.793008    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:46:53.806303 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.806275    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:53.903896 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.903870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.904014 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.903901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:53.904076 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:53.904037    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:46:53.904076 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:53.904053    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 20:46:53.904149 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:53.904104    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm podName:f4ad4488-3e80-4ce5-84c3-1b6524053abe nodeName:}" failed. No retries permitted until 2026-07-01 20:46:54.904088636 +0000 UTC m=+47.815423249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sk28n" (UID: "f4ad4488-3e80-4ce5-84c3-1b6524053abe") : configmap "kube-rbac-proxy" not found
Jul 01 20:46:53.904149 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:53.904117    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls podName:f4ad4488-3e80-4ce5-84c3-1b6524053abe nodeName:}" failed. No retries permitted until 2026-07-01 20:46:54.904111495 +0000 UTC m=+47.815446099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sk28n" (UID: "f4ad4488-3e80-4ce5-84c3-1b6524053abe") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:46:53.934529 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.934501    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb" event={"ID":"b484c66a-201d-48bd-9925-25bc2c2170af","Type":"ContainerStarted","Data":"bcda6be965475179df335c1a9b2bc1b689883e0bab84a93d4481b17fd32038f5"}
Jul 01 20:46:53.936051 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.936036    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/1.log"
Jul 01 20:46:53.936365 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:53.936351    2573 scope.go:117] "RemoveContainer" containerID="372189b8d38d3890b737a6c8d2e89cdb08bc70e373cd44446bc0b038ee2c733b"
Jul 01 20:46:53.936520 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:53.936501    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-r5d2x_openshift-console-operator(4861b515-af80-46a0-a870-438254fa8c38)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" podUID="4861b515-af80-46a0-a870-438254fa8c38"
Jul 01 20:46:54.307363 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:54.307331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:54.307522 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:54.307460    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:46:54.307522 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:54.307513    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert podName:3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:58.307499788 +0000 UTC m=+51.218834392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-fhkqt" (UID: "3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0") : secret "networking-console-plugin-cert" not found
Jul 01 20:46:54.912630 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:54.912593    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:54.912630 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:54.912636    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:54.912880 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:54.912746    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:46:54.912880 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:54.912784    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 20:46:54.912880 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:54.912809    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls podName:f4ad4488-3e80-4ce5-84c3-1b6524053abe nodeName:}" failed. No retries permitted until 2026-07-01 20:46:56.912792987 +0000 UTC m=+49.824127590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sk28n" (UID: "f4ad4488-3e80-4ce5-84c3-1b6524053abe") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:46:54.912880 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:54.912839    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm podName:f4ad4488-3e80-4ce5-84c3-1b6524053abe nodeName:}" failed. No retries permitted until 2026-07-01 20:46:56.912822641 +0000 UTC m=+49.824157247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sk28n" (UID: "f4ad4488-3e80-4ce5-84c3-1b6524053abe") : configmap "kube-rbac-proxy" not found
Jul 01 20:46:54.939848 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:54.939824    2573 scope.go:117] "RemoveContainer" containerID="372189b8d38d3890b737a6c8d2e89cdb08bc70e373cd44446bc0b038ee2c733b"
Jul 01 20:46:54.940190 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:54.939999    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-r5d2x_openshift-console-operator(4861b515-af80-46a0-a870-438254fa8c38)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" podUID="4861b515-af80-46a0-a870-438254fa8c38"
Jul 01 20:46:55.131085 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.131056    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-llqwv_f4938d98-b1e1-4724-97e2-0e948765bd87/dns-node-resolver/0.log"
Jul 01 20:46:55.279588 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.279563    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"]
Jul 01 20:46:55.308485 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.308465    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"]
Jul 01 20:46:55.308579 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.308542    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.311735 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.311716    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 01 20:46:55.311822 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.311716    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 01 20:46:55.312754 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.312735    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fzkfl\""
Jul 01 20:46:55.312889 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.312813    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 01 20:46:55.312889 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.312835    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 01 20:46:55.316839 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.316816    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:55.416834 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.416805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/890d19e1-8364-48a1-b8e8-c83e37f404d4-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-6bnkj\" (UID: \"890d19e1-8364-48a1-b8e8-c83e37f404d4\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.416957 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.416933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknk7\" (UniqueName: \"kubernetes.io/projected/890d19e1-8364-48a1-b8e8-c83e37f404d4-kube-api-access-tknk7\") pod \"service-ca-7bb4f4bcc7-6bnkj\" (UID: \"890d19e1-8364-48a1-b8e8-c83e37f404d4\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.417080 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.417062    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/890d19e1-8364-48a1-b8e8-c83e37f404d4-signing-key\") pod \"service-ca-7bb4f4bcc7-6bnkj\" (UID: \"890d19e1-8364-48a1-b8e8-c83e37f404d4\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.417805 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.417785    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:55.518292 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.518232    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknk7\" (UniqueName: \"kubernetes.io/projected/890d19e1-8364-48a1-b8e8-c83e37f404d4-kube-api-access-tknk7\") pod \"service-ca-7bb4f4bcc7-6bnkj\" (UID: \"890d19e1-8364-48a1-b8e8-c83e37f404d4\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.518401 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.518315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/890d19e1-8364-48a1-b8e8-c83e37f404d4-signing-key\") pod \"service-ca-7bb4f4bcc7-6bnkj\" (UID: \"890d19e1-8364-48a1-b8e8-c83e37f404d4\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.518401 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.518375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/890d19e1-8364-48a1-b8e8-c83e37f404d4-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-6bnkj\" (UID: \"890d19e1-8364-48a1-b8e8-c83e37f404d4\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.519012 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.518982    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/890d19e1-8364-48a1-b8e8-c83e37f404d4-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-6bnkj\" (UID: \"890d19e1-8364-48a1-b8e8-c83e37f404d4\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.520528 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.520507    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/890d19e1-8364-48a1-b8e8-c83e37f404d4-signing-key\") pod \"service-ca-7bb4f4bcc7-6bnkj\" (UID: \"890d19e1-8364-48a1-b8e8-c83e37f404d4\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.526954 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.526931    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknk7\" (UniqueName: \"kubernetes.io/projected/890d19e1-8364-48a1-b8e8-c83e37f404d4-kube-api-access-tknk7\") pod \"service-ca-7bb4f4bcc7-6bnkj\" (UID: \"890d19e1-8364-48a1-b8e8-c83e37f404d4\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.617323 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.617287    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"
Jul 01 20:46:55.755083 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.755056    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj"]
Jul 01 20:46:55.757949 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:46:55.757927    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890d19e1_8364_48a1_b8e8_c83e37f404d4.slice/crio-a860353e0d62d48a6f3c5c1289e89fbe611e33b0bfe3f70d0c213ae485daab0e WatchSource:0}: Error finding container a860353e0d62d48a6f3c5c1289e89fbe611e33b0bfe3f70d0c213ae485daab0e: Status 404 returned error can't find the container with id a860353e0d62d48a6f3c5c1289e89fbe611e33b0bfe3f70d0c213ae485daab0e
Jul 01 20:46:55.931438 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.931413    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ffv5j_634951d8-071a-4335-9c1d-ef274d90307e/node-ca/0.log"
Jul 01 20:46:55.943523 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.943494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj" event={"ID":"890d19e1-8364-48a1-b8e8-c83e37f404d4","Type":"ContainerStarted","Data":"ac3c3b492d9e4ac0b7d3c4570c1a8637466143526f47d35334641a05affceb27"}
Jul 01 20:46:55.943920 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.943543    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj" event={"ID":"890d19e1-8364-48a1-b8e8-c83e37f404d4","Type":"ContainerStarted","Data":"a860353e0d62d48a6f3c5c1289e89fbe611e33b0bfe3f70d0c213ae485daab0e"}
Jul 01 20:46:55.945242 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.945222    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb" event={"ID":"b484c66a-201d-48bd-9925-25bc2c2170af","Type":"ContainerStarted","Data":"dd9c0748ac1b57fbbc55215d659c2d35c3a71e24dd9d4dfde520ae677a26374e"}
Jul 01 20:46:55.945315 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.945261    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb" event={"ID":"b484c66a-201d-48bd-9925-25bc2c2170af","Type":"ContainerStarted","Data":"5ec7665d0985a5dfe9a0e579a409d7b209997703604f58f1fd8eeddcb880ed0b"}
Jul 01 20:46:55.967285 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.967245    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-6bnkj" podStartSLOduration=0.96723265 podStartE2EDuration="967.23265ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:55.966177318 +0000 UTC m=+48.877511944" watchObservedRunningTime="2026-07-01 20:46:55.96723265 +0000 UTC m=+48.878567275"
Jul 01 20:46:55.994597 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:55.994558    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-2hghb" podStartSLOduration=2.060708043 podStartE2EDuration="3.994546239s" totalImagesPullingTime="1.933838196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:55.993801796 +0000 UTC m=+48.905136423" watchObservedRunningTime="2026-07-01 20:46:55.994546239 +0000 UTC m=+48.905880906"
Jul 01 20:46:56.286470 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.286443    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:56.389588 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.389556    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:56.474751 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.474718    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:56.587564 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.587530    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:56.680416 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.680388    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:56.731166 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.731138    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s6d8g_90dd030d-72b9-4b93-ae9b-24aeec54eca6/serve-healthcheck-canary/0.log"
Jul 01 20:46:56.931880 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.931594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:56.931880 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.931642    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:46:56.931880 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:56.931780    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:46:56.931880 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:56.931815    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 20:46:56.931880 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:56.931880    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls podName:f4ad4488-3e80-4ce5-84c3-1b6524053abe nodeName:}" failed. No retries permitted until 2026-07-01 20:47:00.93184705 +0000 UTC m=+53.843181669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sk28n" (UID: "f4ad4488-3e80-4ce5-84c3-1b6524053abe") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:46:56.932244 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:56.931903    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm podName:f4ad4488-3e80-4ce5-84c3-1b6524053abe nodeName:}" failed. No retries permitted until 2026-07-01 20:47:00.931890975 +0000 UTC m=+53.843225587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sk28n" (UID: "f4ad4488-3e80-4ce5-84c3-1b6524053abe") : configmap "kube-rbac-proxy" not found
Jul 01 20:46:56.936441 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.936416    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-2hghb_b484c66a-201d-48bd-9925-25bc2c2170af/migrator/0.log"
Jul 01 20:46:56.967551 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:56.967521    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:57.108723 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:57.108691    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log" limit=1024
Jul 01 20:46:57.133091 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:57.133060    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-2hghb_b484c66a-201d-48bd-9925-25bc2c2170af/graceful-termination/0.log"
Jul 01 20:46:57.340196 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:57.340171    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-6j4dv_e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc/kube-storage-version-migrator-operator/0.log"
Jul 01 20:46:58.341627 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:58.341584    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:46:58.342034 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:58.341733    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 20:46:58.342034 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:58.341797    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert podName:3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:06.341782542 +0000 UTC m=+59.253117149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-fhkqt" (UID: "3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0") : secret "networking-console-plugin-cert" not found
Jul 01 20:46:58.745144 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:58.745060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:46:58.745277 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:58.745204    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 20:46:58.745277 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:58.745220    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74756dfdf5-52bcr: secret "image-registry-tls" not found
Jul 01 20:46:58.745277 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:58.745225    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:46:58.745277 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:58.745269    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls podName:08e4dcd0-b1bf-4ec9-a379-68343eec3f23 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:14.745254843 +0000 UTC m=+67.656589450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls") pod "image-registry-74756dfdf5-52bcr" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23") : secret "image-registry-tls" not found
Jul 01 20:46:58.745484 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:58.745321    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 20:46:58.745484 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:58.745371    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls podName:f8705dc8-bdeb-4766-b015-b5b864ca8ae7 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:14.745358432 +0000 UTC m=+67.656693040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls") pod "dns-default-fj468" (UID: "f8705dc8-bdeb-4766-b015-b5b864ca8ae7") : secret "dns-default-metrics-tls" not found
Jul 01 20:46:59.047132 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:59.047054    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:46:59.047268 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:59.047203    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:59.047268 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:59.047260    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls podName:75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d nodeName:}" failed. No retries permitted until 2026-07-01 20:47:15.047244621 +0000 UTC m=+67.958579234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-6xhzb" (UID: "75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d") : secret "cluster-monitoring-operator-tls" not found
Jul 01 20:46:59.148271 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:59.148243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:59.148405 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:59.148315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:46:59.148405 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:59.148380    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 20:46:59.148483 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:59.148431    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:47:15.148419712 +0000 UTC m=+68.059754320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : configmap references non-existent config key: service-ca.crt
Jul 01 20:46:59.148483 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:46:59.148445    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs podName:1561f8b3-58e2-428a-98e1-5e2d115dd1ff nodeName:}" failed. No retries permitted until 2026-07-01 20:47:15.148438661 +0000 UTC m=+68.059773269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs") pod "router-default-bcd57767c-rwpfk" (UID: "1561f8b3-58e2-428a-98e1-5e2d115dd1ff") : secret "router-metrics-certs-default" not found
Jul 01 20:46:59.249119 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:59.249093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:59.251379 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:59.251347    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f70f9a-122e-4fbf-85e1-9d6db6059227-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9lfqm\" (UID: \"c7f70f9a-122e-4fbf-85e1-9d6db6059227\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:59.378438 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:59.378405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"
Jul 01 20:46:59.532211 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:59.532146    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm"]
Jul 01 20:46:59.958908 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:46:59.958876    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm" event={"ID":"c7f70f9a-122e-4fbf-85e1-9d6db6059227","Type":"ContainerStarted","Data":"e62552c43f8d53d98b0306b652883947da0cffd6eb6131bb592ecb98a00ab1f3"}
Jul 01 20:47:00.965204 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:00.965165    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:47:00.965654 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:00.965223    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:47:00.965654 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:47:00.965316    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:00.965654 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:47:00.965381    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls podName:f4ad4488-3e80-4ce5-84c3-1b6524053abe nodeName:}" failed. No retries permitted until 2026-07-01 20:47:08.965361095 +0000 UTC m=+61.876695711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sk28n" (UID: "f4ad4488-3e80-4ce5-84c3-1b6524053abe") : secret "insights-runtime-extractor-tls" not found
Jul 01 20:47:00.965811 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:00.965788    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ad4488-3e80-4ce5-84c3-1b6524053abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:47:01.971613 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:01.971575    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm" event={"ID":"c7f70f9a-122e-4fbf-85e1-9d6db6059227","Type":"ContainerStarted","Data":"4ee19155e2982961f5bb478d4d7fda188644a945f960cdd7c9ded432979b45db"}
Jul 01 20:47:01.972022 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:01.971646    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm" event={"ID":"c7f70f9a-122e-4fbf-85e1-9d6db6059227","Type":"ContainerStarted","Data":"8cbb6ecb4bfec73fcdbcb22203e49327fda24bb8bf137abdaafe012a15d30e5e"}
Jul 01 20:47:01.988434 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:01.988393    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9lfqm" podStartSLOduration=17.138686466 podStartE2EDuration="18.988379201s" totalImagesPullingTime="1.849692735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:01.987833053 +0000 UTC m=+54.899167677" watchObservedRunningTime="2026-07-01 20:47:01.988379201 +0000 UTC m=+54.899713827"
Jul 01 20:47:06.412656 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:06.412620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:47:06.415090 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:06.415070    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-fhkqt\" (UID: \"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:47:06.550683 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:06.550651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"
Jul 01 20:47:06.707192 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:06.707171    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt"]
Jul 01 20:47:06.709992 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:06.709958    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef16bee_7f4f_43cf_a467_ea42c7b6b2a0.slice/crio-492075a409df031bf362b9dc8f1dd66bdcfcec972a26837944f5be041870859c WatchSource:0}: Error finding container 492075a409df031bf362b9dc8f1dd66bdcfcec972a26837944f5be041870859c: Status 404 returned error can't find the container with id 492075a409df031bf362b9dc8f1dd66bdcfcec972a26837944f5be041870859c
Jul 01 20:47:06.988420 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:06.988353    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt" event={"ID":"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0","Type":"ContainerStarted","Data":"492075a409df031bf362b9dc8f1dd66bdcfcec972a26837944f5be041870859c"}
Jul 01 20:47:07.836495 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:07.836468    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lrvqz"
Jul 01 20:47:08.585642 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:08.585613    2573 scope.go:117] "RemoveContainer" containerID="372189b8d38d3890b737a6c8d2e89cdb08bc70e373cd44446bc0b038ee2c733b"
Jul 01 20:47:08.997528 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:08.997458    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 20:47:08.997959 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:08.997941    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/1.log"
Jul 01 20:47:08.998044 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:08.997986    2573 generic.go:356] "Generic (PLEG): container finished" podID="4861b515-af80-46a0-a870-438254fa8c38" containerID="8d51d822d4d84dd1b6ca4988a27aa59fc63cf82f3b0e4ed27ca3b33d7b81211a" exitCode=255
Jul 01 20:47:08.998099 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:08.998055    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" event={"ID":"4861b515-af80-46a0-a870-438254fa8c38","Type":"ContainerDied","Data":"8d51d822d4d84dd1b6ca4988a27aa59fc63cf82f3b0e4ed27ca3b33d7b81211a"}
Jul 01 20:47:08.998153 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:08.998107    2573 scope.go:117] "RemoveContainer" containerID="372189b8d38d3890b737a6c8d2e89cdb08bc70e373cd44446bc0b038ee2c733b"
Jul 01 20:47:08.998422 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:08.998399    2573 scope.go:117] "RemoveContainer" containerID="8d51d822d4d84dd1b6ca4988a27aa59fc63cf82f3b0e4ed27ca3b33d7b81211a"
Jul 01 20:47:08.998593 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:47:08.998574    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-r5d2x_openshift-console-operator(4861b515-af80-46a0-a870-438254fa8c38)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" podUID="4861b515-af80-46a0-a870-438254fa8c38"
Jul 01 20:47:08.999798 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:08.999773    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt" event={"ID":"3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0","Type":"ContainerStarted","Data":"d65c3a9670087ed057ab63174f9bb0ee2c7d18172a550841dff42559d4f1ac09"}
Jul 01 20:47:09.035590 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:09.035563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:47:09.037057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:09.037017    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-fhkqt" podStartSLOduration=17.784855221 podStartE2EDuration="19.037002019s" totalImagesPullingTime="1.252146798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:09.036276517 +0000 UTC m=+61.947611144" watchObservedRunningTime="2026-07-01 20:47:09.037002019 +0000 UTC m=+61.948336644"
Jul 01 20:47:09.037932 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:09.037908    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ad4488-3e80-4ce5-84c3-1b6524053abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sk28n\" (UID: \"f4ad4488-3e80-4ce5-84c3-1b6524053abe\") " pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:47:09.069973 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:09.069953    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9fsnr\""
Jul 01 20:47:09.077441 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:09.077426    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sk28n"
Jul 01 20:47:09.202596 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:09.202569    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sk28n"]
Jul 01 20:47:09.206013 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:09.205986    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ad4488_3e80_4ce5_84c3_1b6524053abe.slice/crio-432feb231205e61b64dc8ef8ec8ac79bf4feadf185bd2143f5e6c54cd01055d2 WatchSource:0}: Error finding container 432feb231205e61b64dc8ef8ec8ac79bf4feadf185bd2143f5e6c54cd01055d2: Status 404 returned error can't find the container with id 432feb231205e61b64dc8ef8ec8ac79bf4feadf185bd2143f5e6c54cd01055d2
Jul 01 20:47:10.006558 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:10.006516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sk28n" event={"ID":"f4ad4488-3e80-4ce5-84c3-1b6524053abe","Type":"ContainerStarted","Data":"432feb231205e61b64dc8ef8ec8ac79bf4feadf185bd2143f5e6c54cd01055d2"}
Jul 01 20:47:10.009155 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:10.008610    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 20:47:13.021111 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.021084    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sk28n" event={"ID":"f4ad4488-3e80-4ce5-84c3-1b6524053abe","Type":"ContainerStarted","Data":"c0680faa8aa007e6d2974aade8f2c31daec635115b07141df3af160ae93e66d2"}
Jul 01 20:47:13.375344 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.375316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:47:13.378833 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.378815    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 20:47:13.388541 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.388517    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb437fb-681b-464b-a74b-e670592d6802-metrics-certs\") pod \"network-metrics-daemon-sqdk2\" (UID: \"deb437fb-681b-464b-a74b-e670592d6802\") " pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:47:13.403633 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.403592    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v6s46\""
Jul 01 20:47:13.411810 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.411791    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqdk2"
Jul 01 20:47:13.477468 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.477433    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:47:13.480606 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.480581    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 20:47:13.491426 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.491367    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eb902c9-58f4-4714-b5c4-3259d67349f9-original-pull-secret\") pod \"global-pull-secret-syncer-zwrwf\" (UID: \"6eb902c9-58f4-4714-b5c4-3259d67349f9\") " pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:47:13.556662 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.556637    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sqdk2"]
Jul 01 20:47:13.578003 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.577980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:47:13.580521 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.580498    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2874w\" (UniqueName: \"kubernetes.io/projected/65bb81b4-c408-47ed-a0bb-9798a25206f6-kube-api-access-2874w\") pod \"network-check-target-wzfcf\" (UID: \"65bb81b4-c408-47ed-a0bb-9798a25206f6\") " pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:47:13.655044 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:13.654987    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb437fb_681b_464b_a74b_e670592d6802.slice/crio-8d91610e59cd59c9dc17b97cf2a56d09ffcea799c087232e8b4d252d5dbcbba7 WatchSource:0}: Error finding container 8d91610e59cd59c9dc17b97cf2a56d09ffcea799c087232e8b4d252d5dbcbba7: Status 404 returned error can't find the container with id 8d91610e59cd59c9dc17b97cf2a56d09ffcea799c087232e8b4d252d5dbcbba7
Jul 01 20:47:13.697948 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.697927    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4g6n6\""
Jul 01 20:47:13.705300 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.705284    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:47:13.705416 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.705400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zwrwf"
Jul 01 20:47:13.792747 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.792666    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:47:13.792747 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.792733    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:47:13.793236 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.793169    2573 scope.go:117] "RemoveContainer" containerID="8d51d822d4d84dd1b6ca4988a27aa59fc63cf82f3b0e4ed27ca3b33d7b81211a"
Jul 01 20:47:13.793427 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:47:13.793339    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-r5d2x_openshift-console-operator(4861b515-af80-46a0-a870-438254fa8c38)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" podUID="4861b515-af80-46a0-a870-438254fa8c38"
Jul 01 20:47:13.861057 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.860885    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wzfcf"]
Jul 01 20:47:13.863637 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:13.863614    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bb81b4_c408_47ed_a0bb_9798a25206f6.slice/crio-5985f9e4b2d9f2fc3cbf554308bb2c4d534d330698629ac8959698d045499693 WatchSource:0}: Error finding container 5985f9e4b2d9f2fc3cbf554308bb2c4d534d330698629ac8959698d045499693: Status 404 returned error can't find the container with id 5985f9e4b2d9f2fc3cbf554308bb2c4d534d330698629ac8959698d045499693
Jul 01 20:47:13.893192 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:13.893170    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zwrwf"]
Jul 01 20:47:14.025792 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:14.025736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sqdk2" event={"ID":"deb437fb-681b-464b-a74b-e670592d6802","Type":"ContainerStarted","Data":"8d91610e59cd59c9dc17b97cf2a56d09ffcea799c087232e8b4d252d5dbcbba7"}
Jul 01 20:47:14.026987 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:14.026961    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wzfcf" event={"ID":"65bb81b4-c408-47ed-a0bb-9798a25206f6","Type":"ContainerStarted","Data":"5985f9e4b2d9f2fc3cbf554308bb2c4d534d330698629ac8959698d045499693"}
Jul 01 20:47:14.027890 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:14.027873    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb902c9_58f4_4714_b5c4_3259d67349f9.slice/crio-8ceeb364572e62e4c1b90c520a57cf5e2e3aecc917928508c7c66580a2106e6a WatchSource:0}: Error finding container 8ceeb364572e62e4c1b90c520a57cf5e2e3aecc917928508c7c66580a2106e6a: Status 404 returned error can't find the container with id 8ceeb364572e62e4c1b90c520a57cf5e2e3aecc917928508c7c66580a2106e6a
Jul 01 20:47:14.788105 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:14.788071    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:47:14.788266 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:14.788129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:47:14.790922 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:14.790890    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"image-registry-74756dfdf5-52bcr\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") " pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:47:14.791452 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:14.791426    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8705dc8-bdeb-4766-b015-b5b864ca8ae7-metrics-tls\") pod \"dns-default-fj468\" (UID: \"f8705dc8-bdeb-4766-b015-b5b864ca8ae7\") " pod="openshift-dns/dns-default-fj468"
Jul 01 20:47:15.035110 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.035071    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sk28n" event={"ID":"f4ad4488-3e80-4ce5-84c3-1b6524053abe","Type":"ContainerStarted","Data":"039d6d15ef9870c3ba5c4bf254ce73e14f1679c7bc7c5f7f3a1ff449180be601"}
Jul 01 20:47:15.037355 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.037323    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wzfcf" event={"ID":"65bb81b4-c408-47ed-a0bb-9798a25206f6","Type":"ContainerStarted","Data":"a846e5263e76fa43fe79c80a4bf4920c1c8f7c2fb4c8adfc532f7f8bb12ccc0b"}
Jul 01 20:47:15.037473 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.037409    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:47:15.039405 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.039329    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zwrwf" event={"ID":"6eb902c9-58f4-4714-b5c4-3259d67349f9","Type":"ContainerStarted","Data":"8ceeb364572e62e4c1b90c520a57cf5e2e3aecc917928508c7c66580a2106e6a"}
Jul 01 20:47:15.040934 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.040838    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-txm48\""
Jul 01 20:47:15.048517 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.048487    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:47:15.050831 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.050659    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f5twz\""
Jul 01 20:47:15.060654 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.060601    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wzfcf" podStartSLOduration=67.06058617 podStartE2EDuration="1m7.06058617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:15.060192404 +0000 UTC m=+67.971527030" watchObservedRunningTime="2026-07-01 20:47:15.06058617 +0000 UTC m=+67.971920797"
Jul 01 20:47:15.062439 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.062227    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fj468"
Jul 01 20:47:15.091336 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.090866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:47:15.095675 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.095604    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-6xhzb\" (UID: \"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:47:15.191834 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.191794    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:47:15.192000 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.191913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:47:15.194386 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.193596    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-service-ca-bundle\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:47:15.197213 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.197192    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1561f8b3-58e2-428a-98e1-5e2d115dd1ff-metrics-certs\") pod \"router-default-bcd57767c-rwpfk\" (UID: \"1561f8b3-58e2-428a-98e1-5e2d115dd1ff\") " pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:47:15.234395 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.233928    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74756dfdf5-52bcr"]
Jul 01 20:47:15.251568 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.251496    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fj468"]
Jul 01 20:47:15.354350 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.354320    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8jb8c\""
Jul 01 20:47:15.361295 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.361202    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"
Jul 01 20:47:15.438950 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:15.438913    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e4dcd0_b1bf_4ec9_a379_68343eec3f23.slice/crio-7bbf1852aec5706e73e05786398dd0f0b48d10141767f07e205fd8580cb5351d WatchSource:0}: Error finding container 7bbf1852aec5706e73e05786398dd0f0b48d10141767f07e205fd8580cb5351d: Status 404 returned error can't find the container with id 7bbf1852aec5706e73e05786398dd0f0b48d10141767f07e205fd8580cb5351d
Jul 01 20:47:15.440604 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:15.440569    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8705dc8_bdeb_4766_b015_b5b864ca8ae7.slice/crio-2ef084624b04c126ed956ec27d4d5c0ae81e092e4486251cb9142c190d4aff0b WatchSource:0}: Error finding container 2ef084624b04c126ed956ec27d4d5c0ae81e092e4486251cb9142c190d4aff0b: Status 404 returned error can't find the container with id 2ef084624b04c126ed956ec27d4d5c0ae81e092e4486251cb9142c190d4aff0b
Jul 01 20:47:15.442791 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.442768    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f88nq\""
Jul 01 20:47:15.450413 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.450359    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:47:15.620488 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.620441    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb"]
Jul 01 20:47:15.622813 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:15.622780    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c5a5b9_1fc9_4382_bf2e_ff152ad5a70d.slice/crio-4d794da502ba7afa52b08c3a501a24696313b8adcbc2111d7348694c9a8ad4b2 WatchSource:0}: Error finding container 4d794da502ba7afa52b08c3a501a24696313b8adcbc2111d7348694c9a8ad4b2: Status 404 returned error can't find the container with id 4d794da502ba7afa52b08c3a501a24696313b8adcbc2111d7348694c9a8ad4b2
Jul 01 20:47:15.660042 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:15.658245    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bcd57767c-rwpfk"]
Jul 01 20:47:15.664160 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:15.664120    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1561f8b3_58e2_428a_98e1_5e2d115dd1ff.slice/crio-67e8b5d5f86c9387b2354e64f35bf1d6432e2b76d5c20d2ef604ab9f356f107c WatchSource:0}: Error finding container 67e8b5d5f86c9387b2354e64f35bf1d6432e2b76d5c20d2ef604ab9f356f107c: Status 404 returned error can't find the container with id 67e8b5d5f86c9387b2354e64f35bf1d6432e2b76d5c20d2ef604ab9f356f107c
Jul 01 20:47:16.048292 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.048171    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fj468" event={"ID":"f8705dc8-bdeb-4766-b015-b5b864ca8ae7","Type":"ContainerStarted","Data":"2ef084624b04c126ed956ec27d4d5c0ae81e092e4486251cb9142c190d4aff0b"}
Jul 01 20:47:16.052808 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.052250    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sqdk2" event={"ID":"deb437fb-681b-464b-a74b-e670592d6802","Type":"ContainerStarted","Data":"babf3554f038027b816718246bf742ac77915fe758d198220c9411b833a267ab"}
Jul 01 20:47:16.052808 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.052304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sqdk2" event={"ID":"deb437fb-681b-464b-a74b-e670592d6802","Type":"ContainerStarted","Data":"1e8f449a5189d9c6bdb9cd50e754ea1f2c53eec2c4df6a4c475e291e3e2c4036"}
Jul 01 20:47:16.054238 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.054211    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bcd57767c-rwpfk" event={"ID":"1561f8b3-58e2-428a-98e1-5e2d115dd1ff","Type":"ContainerStarted","Data":"4c6af2f8f4d9bbe806f238c458aaeb252bf71ce76a8c693e33351ecf978385ba"}
Jul 01 20:47:16.054357 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.054256    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bcd57767c-rwpfk" event={"ID":"1561f8b3-58e2-428a-98e1-5e2d115dd1ff","Type":"ContainerStarted","Data":"67e8b5d5f86c9387b2354e64f35bf1d6432e2b76d5c20d2ef604ab9f356f107c"}
Jul 01 20:47:16.058542 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.058437    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr" event={"ID":"08e4dcd0-b1bf-4ec9-a379-68343eec3f23","Type":"ContainerStarted","Data":"8dc3dc2ff036114f2ce06fed2e29cf9f5577b6aba7da39d1ad0070f3f067d8ef"}
Jul 01 20:47:16.058542 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.058477    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr" event={"ID":"08e4dcd0-b1bf-4ec9-a379-68343eec3f23","Type":"ContainerStarted","Data":"7bbf1852aec5706e73e05786398dd0f0b48d10141767f07e205fd8580cb5351d"}
Jul 01 20:47:16.058542 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.058502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:47:16.061015 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.060971    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb" event={"ID":"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d","Type":"ContainerStarted","Data":"4d794da502ba7afa52b08c3a501a24696313b8adcbc2111d7348694c9a8ad4b2"}
Jul 01 20:47:16.071248 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.071200    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sqdk2" podStartSLOduration=66.250502307 podStartE2EDuration="1m8.07118449s" totalImagesPullingTime="1.820682183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:16.070877656 +0000 UTC m=+68.982212282" watchObservedRunningTime="2026-07-01 20:47:16.07118449 +0000 UTC m=+68.982519117"
Jul 01 20:47:16.096624 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.095734    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr" podStartSLOduration=67.095715892 podStartE2EDuration="1m7.095715892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:16.09442147 +0000 UTC m=+69.005756100" watchObservedRunningTime="2026-07-01 20:47:16.095715892 +0000 UTC m=+69.007050522"
Jul 01 20:47:16.450838 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.450803    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:47:16.453674 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.453646    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:47:16.475123 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:16.475064    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-bcd57767c-rwpfk" podStartSLOduration=33.475043136 podStartE2EDuration="33.475043136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:16.119253923 +0000 UTC m=+69.030588551" watchObservedRunningTime="2026-07-01 20:47:16.475043136 +0000 UTC m=+69.386377763"
Jul 01 20:47:17.068415 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:17.068334    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:47:17.070284 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:17.070051    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-bcd57767c-rwpfk"
Jul 01 20:47:20.083537 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.083484    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sk28n" event={"ID":"f4ad4488-3e80-4ce5-84c3-1b6524053abe","Type":"ContainerStarted","Data":"42ac361584af09e9081a585e26286f389b888dd5042718a52f79d0b4caea708e"}
Jul 01 20:47:20.085690 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.085662    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fj468" event={"ID":"f8705dc8-bdeb-4766-b015-b5b864ca8ae7","Type":"ContainerStarted","Data":"05d5b3d80b920b3358169ee018f28b1953ee4950dec1912107d3c03eee2d388b"}
Jul 01 20:47:20.085796 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.085728    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fj468" event={"ID":"f8705dc8-bdeb-4766-b015-b5b864ca8ae7","Type":"ContainerStarted","Data":"b616758b31fcaedc1cfd593b34929a430f7e6102886fbafe92ca0e97ff40ac48"}
Jul 01 20:47:20.086373 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.086356    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fj468"
Jul 01 20:47:20.087990 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.087964    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb" event={"ID":"75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d","Type":"ContainerStarted","Data":"44467ebf8a61d4f98ca0963a742bc9f161e05b34e5ab28db35c9807507084333"}
Jul 01 20:47:20.089552 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.089528    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zwrwf" event={"ID":"6eb902c9-58f4-4714-b5c4-3259d67349f9","Type":"ContainerStarted","Data":"c42c87022078f6884a3470af40aeb9b36eac126493435ff35c855f260da5617b"}
Jul 01 20:47:20.106402 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.106358    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sk28n" podStartSLOduration=16.658140169 podStartE2EDuration="27.106341018s" totalImagesPullingTime="10.448200849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:20.104689144 +0000 UTC m=+73.016023768" watchObservedRunningTime="2026-07-01 20:47:20.106341018 +0000 UTC m=+73.017675645"
Jul 01 20:47:20.126389 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.126335    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-6xhzb" podStartSLOduration=32.91191396 podStartE2EDuration="37.126322118s" totalImagesPullingTime="4.214408158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:20.125739573 +0000 UTC m=+73.037074201" watchObservedRunningTime="2026-07-01 20:47:20.126322118 +0000 UTC m=+73.037656723"
Jul 01 20:47:20.145742 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.145701    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zwrwf" podStartSLOduration=65.353978698 podStartE2EDuration="1m11.145689791s" totalImagesPullingTime="5.791711093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:20.143736304 +0000 UTC m=+73.055070930" watchObservedRunningTime="2026-07-01 20:47:20.145689791 +0000 UTC m=+73.057024417"
Jul 01 20:47:20.169009 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.168973    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fj468" podStartSLOduration=33.773131239 podStartE2EDuration="38.168954356s" totalImagesPullingTime="4.395823117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:20.167840466 +0000 UTC m=+73.079175094" watchObservedRunningTime="2026-07-01 20:47:20.168954356 +0000 UTC m=+73.080288979"
Jul 01 20:47:20.650751 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.650718    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"]
Jul 01 20:47:20.679489 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.679460    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"]
Jul 01 20:47:20.679627 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.679604    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"
Jul 01 20:47:20.683418 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.683290    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 20:47:20.683543 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.683474    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pt4lc\""
Jul 01 20:47:20.683543 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.683516    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 20:47:20.683665 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.683476    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 01 20:47:20.683839 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.683819    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 20:47:20.695396 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.695378    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.695396 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.695388    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"]
Jul 01 20:47:20.698221 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.698204    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 20:47:20.698409 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.698395    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 20:47:20.698468 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.698425    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 20:47:20.698510 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.698426    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 20:47:20.713667 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.713651    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"]
Jul 01 20:47:20.713741 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.713676    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"]
Jul 01 20:47:20.713741 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.713687    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"]
Jul 01 20:47:20.713809 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.713759    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:20.730585 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.730567    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 20:47:20.772338 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.772318    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"]
Jul 01 20:47:20.794017 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.793994    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"]
Jul 01 20:47:20.794116 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.794103    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"
Jul 01 20:47:20.797693 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.797674    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-54vzz\""
Jul 01 20:47:20.797794 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.797742    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 01 20:47:20.841068 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh5qs\" (UniqueName: \"kubernetes.io/projected/b97e56d8-0628-41a3-b6ba-dca6f03d03a1-kube-api-access-vh5qs\") pod \"klusterlet-addon-workmgr-5ff48d74cb-nx9v2\" (UID: \"b97e56d8-0628-41a3-b6ba-dca6f03d03a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:20.841169 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841090    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7zr\" (UniqueName: \"kubernetes.io/projected/0ff9a153-622d-42ac-a450-19890f878241-kube-api-access-lk7zr\") pod \"managed-serviceaccount-addon-agent-6bb7484c66-twrnf\" (UID: \"0ff9a153-622d-42ac-a450-19890f878241\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"
Jul 01 20:47:20.841169 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841118    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b97e56d8-0628-41a3-b6ba-dca6f03d03a1-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ff48d74cb-nx9v2\" (UID: \"b97e56d8-0628-41a3-b6ba-dca6f03d03a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:20.841244 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.841244 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.841315 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841260    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c3261b-732d-4dd7-86d0-857c12e617c3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.841315 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841288    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b97e56d8-0628-41a3-b6ba-dca6f03d03a1-tmp\") pod \"klusterlet-addon-workmgr-5ff48d74cb-nx9v2\" (UID: \"b97e56d8-0628-41a3-b6ba-dca6f03d03a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:20.841315 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4g5w\" (UniqueName: \"kubernetes.io/projected/77c3261b-732d-4dd7-86d0-857c12e617c3-kube-api-access-q4g5w\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.841420 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0ff9a153-622d-42ac-a450-19890f878241-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb7484c66-twrnf\" (UID: \"0ff9a153-622d-42ac-a450-19890f878241\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"
Jul 01 20:47:20.841420 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-ca\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.841420 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.841401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-hub\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.942594 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-ca\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.942594 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942569    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-hub\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.942594 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942589    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5qs\" (UniqueName: \"kubernetes.io/projected/b97e56d8-0628-41a3-b6ba-dca6f03d03a1-kube-api-access-vh5qs\") pod \"klusterlet-addon-workmgr-5ff48d74cb-nx9v2\" (UID: \"b97e56d8-0628-41a3-b6ba-dca6f03d03a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:20.942823 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942615    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7zr\" (UniqueName: \"kubernetes.io/projected/0ff9a153-622d-42ac-a450-19890f878241-kube-api-access-lk7zr\") pod \"managed-serviceaccount-addon-agent-6bb7484c66-twrnf\" (UID: \"0ff9a153-622d-42ac-a450-19890f878241\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"
Jul 01 20:47:20.942823 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942643    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b97e56d8-0628-41a3-b6ba-dca6f03d03a1-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ff48d74cb-nx9v2\" (UID: \"b97e56d8-0628-41a3-b6ba-dca6f03d03a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:20.942823 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.942823 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.942823 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c3261b-732d-4dd7-86d0-857c12e617c3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.942823 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942781    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f0c0783c-df02-47db-8df1-0bf67b5d7b19-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-gh4cp\" (UID: \"f0c0783c-df02-47db-8df1-0bf67b5d7b19\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"
Jul 01 20:47:20.942823 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b97e56d8-0628-41a3-b6ba-dca6f03d03a1-tmp\") pod \"klusterlet-addon-workmgr-5ff48d74cb-nx9v2\" (UID: \"b97e56d8-0628-41a3-b6ba-dca6f03d03a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:20.943198 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942834    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4g5w\" (UniqueName: \"kubernetes.io/projected/77c3261b-732d-4dd7-86d0-857c12e617c3-kube-api-access-q4g5w\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.943198 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.942889    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0ff9a153-622d-42ac-a450-19890f878241-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb7484c66-twrnf\" (UID: \"0ff9a153-622d-42ac-a450-19890f878241\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"
Jul 01 20:47:20.943521 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.943467    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b97e56d8-0628-41a3-b6ba-dca6f03d03a1-tmp\") pod \"klusterlet-addon-workmgr-5ff48d74cb-nx9v2\" (UID: \"b97e56d8-0628-41a3-b6ba-dca6f03d03a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:20.943655 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.943622    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c3261b-732d-4dd7-86d0-857c12e617c3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.945643 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.945612    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.945796 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.945771    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-hub\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.945925 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.945901    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0ff9a153-622d-42ac-a450-19890f878241-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb7484c66-twrnf\" (UID: \"0ff9a153-622d-42ac-a450-19890f878241\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"
Jul 01 20:47:20.946002 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.945972    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b97e56d8-0628-41a3-b6ba-dca6f03d03a1-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ff48d74cb-nx9v2\" (UID: \"b97e56d8-0628-41a3-b6ba-dca6f03d03a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:20.946060 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.946044    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.946334 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.946317    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/77c3261b-732d-4dd7-86d0-857c12e617c3-ca\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.956283 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.955937    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4g5w\" (UniqueName: \"kubernetes.io/projected/77c3261b-732d-4dd7-86d0-857c12e617c3-kube-api-access-q4g5w\") pod \"cluster-proxy-proxy-agent-579f755b7-dzz2c\" (UID: \"77c3261b-732d-4dd7-86d0-857c12e617c3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:20.957515 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.957491    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7zr\" (UniqueName: \"kubernetes.io/projected/0ff9a153-622d-42ac-a450-19890f878241-kube-api-access-lk7zr\") pod \"managed-serviceaccount-addon-agent-6bb7484c66-twrnf\" (UID: \"0ff9a153-622d-42ac-a450-19890f878241\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"
Jul 01 20:47:20.958162 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:20.958139    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh5qs\" (UniqueName: \"kubernetes.io/projected/b97e56d8-0628-41a3-b6ba-dca6f03d03a1-kube-api-access-vh5qs\") pod \"klusterlet-addon-workmgr-5ff48d74cb-nx9v2\" (UID: \"b97e56d8-0628-41a3-b6ba-dca6f03d03a1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:21.001026 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.001000    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"
Jul 01 20:47:21.008306 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.008285    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"
Jul 01 20:47:21.039690 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.039663    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:21.043975 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.043953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f0c0783c-df02-47db-8df1-0bf67b5d7b19-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-gh4cp\" (UID: \"f0c0783c-df02-47db-8df1-0bf67b5d7b19\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"
Jul 01 20:47:21.046318 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.046296    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f0c0783c-df02-47db-8df1-0bf67b5d7b19-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-gh4cp\" (UID: \"f0c0783c-df02-47db-8df1-0bf67b5d7b19\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"
Jul 01 20:47:21.103369 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.103152    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"
Jul 01 20:47:21.175881 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.175823    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf"]
Jul 01 20:47:21.179948 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:21.179914    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ff9a153_622d_42ac_a450_19890f878241.slice/crio-f8d355ea8d0475c6664fbea708667e3b5573d9351c336b5681f4d3106db1a425 WatchSource:0}: Error finding container f8d355ea8d0475c6664fbea708667e3b5573d9351c336b5681f4d3106db1a425: Status 404 returned error can't find the container with id f8d355ea8d0475c6664fbea708667e3b5573d9351c336b5681f4d3106db1a425
Jul 01 20:47:21.211735 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.211705    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c"]
Jul 01 20:47:21.214355 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:21.214318    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77c3261b_732d_4dd7_86d0_857c12e617c3.slice/crio-5e09f800bce7e323e1476859f2c3b23c79b904d435a042ad75f17be5122ba25b WatchSource:0}: Error finding container 5e09f800bce7e323e1476859f2c3b23c79b904d435a042ad75f17be5122ba25b: Status 404 returned error can't find the container with id 5e09f800bce7e323e1476859f2c3b23c79b904d435a042ad75f17be5122ba25b
Jul 01 20:47:21.230117 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.228223    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"]
Jul 01 20:47:21.230117 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:21.228494    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97e56d8_0628_41a3_b6ba_dca6f03d03a1.slice/crio-9564e017b7b83206c3b7cde6b357b3aacd326e848a7c4271935362275d3c4fbd WatchSource:0}: Error finding container 9564e017b7b83206c3b7cde6b357b3aacd326e848a7c4271935362275d3c4fbd: Status 404 returned error can't find the container with id 9564e017b7b83206c3b7cde6b357b3aacd326e848a7c4271935362275d3c4fbd
Jul 01 20:47:21.283779 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:21.283758    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"]
Jul 01 20:47:21.284881 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:21.284843    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c0783c_df02_47db_8df1_0bf67b5d7b19.slice/crio-734c6e7e8d0c0605538f1e55dfeaa695e783a868f9edbde8bb3e3f295d7442b9 WatchSource:0}: Error finding container 734c6e7e8d0c0605538f1e55dfeaa695e783a868f9edbde8bb3e3f295d7442b9: Status 404 returned error can't find the container with id 734c6e7e8d0c0605538f1e55dfeaa695e783a868f9edbde8bb3e3f295d7442b9
Jul 01 20:47:22.101216 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:22.101180    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c" event={"ID":"77c3261b-732d-4dd7-86d0-857c12e617c3","Type":"ContainerStarted","Data":"5e09f800bce7e323e1476859f2c3b23c79b904d435a042ad75f17be5122ba25b"}
Jul 01 20:47:22.102259 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:22.102236    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf" event={"ID":"0ff9a153-622d-42ac-a450-19890f878241","Type":"ContainerStarted","Data":"f8d355ea8d0475c6664fbea708667e3b5573d9351c336b5681f4d3106db1a425"}
Jul 01 20:47:22.103327 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:22.103305    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2" event={"ID":"b97e56d8-0628-41a3-b6ba-dca6f03d03a1","Type":"ContainerStarted","Data":"9564e017b7b83206c3b7cde6b357b3aacd326e848a7c4271935362275d3c4fbd"}
Jul 01 20:47:22.104429 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:22.104406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp" event={"ID":"f0c0783c-df02-47db-8df1-0bf67b5d7b19","Type":"ContainerStarted","Data":"734c6e7e8d0c0605538f1e55dfeaa695e783a868f9edbde8bb3e3f295d7442b9"}
Jul 01 20:47:26.586192 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:26.586165    2573 scope.go:117] "RemoveContainer" containerID="8d51d822d4d84dd1b6ca4988a27aa59fc63cf82f3b0e4ed27ca3b33d7b81211a"
Jul 01 20:47:26.586576 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:47:26.586317    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-r5d2x_openshift-console-operator(4861b515-af80-46a0-a870-438254fa8c38)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" podUID="4861b515-af80-46a0-a870-438254fa8c38"
Jul 01 20:47:28.145231 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.145193    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp" event={"ID":"f0c0783c-df02-47db-8df1-0bf67b5d7b19","Type":"ContainerStarted","Data":"f7153c5df8b5f9c9d5a1970817b309036dbd626a8fb20c4e8804999c6904282f"}
Jul 01 20:47:28.146030 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.145262    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"
Jul 01 20:47:28.147707 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.147678    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c" event={"ID":"77c3261b-732d-4dd7-86d0-857c12e617c3","Type":"ContainerStarted","Data":"1a44993330e926bf454421aeb192c9293e82f58503109fcc48ef30a47339a22b"}
Jul 01 20:47:28.147884 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.147725    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c" event={"ID":"77c3261b-732d-4dd7-86d0-857c12e617c3","Type":"ContainerStarted","Data":"91852f969257186fd4bc0a25d44f72db3a84ca34036ad2c1a9a32babc48f8794"}
Jul 01 20:47:28.147884 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.147745    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c" event={"ID":"77c3261b-732d-4dd7-86d0-857c12e617c3","Type":"ContainerStarted","Data":"0621243adff84605e9959bc3b5332c1c5f6102c2d82e69a2017cd4544f1196c8"}
Jul 01 20:47:28.149341 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.149312    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf" event={"ID":"0ff9a153-622d-42ac-a450-19890f878241","Type":"ContainerStarted","Data":"18846a78836df2b174d557db638ff6ffc69d175249b91368b96f6e5013112529"}
Jul 01 20:47:28.150605 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.150585    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp"
Jul 01 20:47:28.150941 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.150922    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2" event={"ID":"b97e56d8-0628-41a3-b6ba-dca6f03d03a1","Type":"ContainerStarted","Data":"4958f40a842daf69febfed5ed3df3bd81024e674bb1e2a4b4c723aeaa1cb8d3c"}
Jul 01 20:47:28.151144 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.151130    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:28.152424 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.152406    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2"
Jul 01 20:47:28.164163 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.164123    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp" podStartSLOduration=1.704890921 podStartE2EDuration="8.164108708s" totalImagesPullingTime="6.459217787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:28.162711145 +0000 UTC m=+81.074045771" watchObservedRunningTime="2026-07-01 20:47:28.164108708 +0000 UTC m=+81.075443325"
Jul 01 20:47:28.181460 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.181423    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff48d74cb-nx9v2" podStartSLOduration=1.6452414480000002 podStartE2EDuration="8.181409312s" totalImagesPullingTime="6.536167864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:28.180438138 +0000 UTC m=+81.091772763" watchObservedRunningTime="2026-07-01 20:47:28.181409312 +0000 UTC m=+81.092743939"
Jul 01 20:47:28.200981 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.200937    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb7484c66-twrnf" podStartSLOduration=1.634882425 podStartE2EDuration="8.200923922s" totalImagesPullingTime="6.566041497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:28.199495971 +0000 UTC m=+81.110830601" watchObservedRunningTime="2026-07-01 20:47:28.200923922 +0000 UTC m=+81.112258547"
Jul 01 20:47:28.223430 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:28.223392    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579f755b7-dzz2c" podStartSLOduration=1.6932781540000001 podStartE2EDuration="8.223380312s" totalImagesPullingTime="6.530102158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:28.22296957 +0000 UTC m=+81.134304196" watchObservedRunningTime="2026-07-01 20:47:28.223380312 +0000 UTC m=+81.134714973"
Jul 01 20:47:31.106584 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:31.106456    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fj468"
Jul 01 20:47:35.053439 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:35.053405    2573 patch_prober.go:28] interesting pod/image-registry-74756dfdf5-52bcr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 01 20:47:35.053799 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:35.053454    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr" podUID="08e4dcd0-b1bf-4ec9-a379-68343eec3f23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:47:36.946136 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:36.946099    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fzt68"]
Jul 01 20:47:36.949943 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:36.949921    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:36.952526 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:36.952497    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 20:47:36.952717 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:36.952702    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 20:47:36.953902 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:36.953885    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dh9nf\""
Jul 01 20:47:36.954002 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:36.953941    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 20:47:36.954002 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:36.953968    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 20:47:37.063284 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.063263    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.063404 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.063300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.063404 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.063333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-sys\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.063404 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.063390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-root\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.063532 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.063413    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42d7t\" (UniqueName: \"kubernetes.io/projected/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-kube-api-access-42d7t\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.063532 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.063463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-textfile\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.063532 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.063497    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-tls\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.063532 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.063521    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-wtmp\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.063670 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.063553    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-metrics-client-ca\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.072225 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.072209    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:47:37.164653 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.164623    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-metrics-client-ca\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.164790 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.164710    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.164926 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.164902    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165023 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.164990    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-sys\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165086 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-root\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165086 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165069    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42d7t\" (UniqueName: \"kubernetes.io/projected/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-kube-api-access-42d7t\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165187 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165106    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-sys\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165187 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165116    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-textfile\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165290 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-tls\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165290 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165237    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-wtmp\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165391 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165296    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-metrics-client-ca\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165391 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165352    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165493 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165385    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-textfile\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165493 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165396    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-wtmp\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.165493 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.165422    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-root\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.167211 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.167191    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.167503 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.167485    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-node-exporter-tls\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.173446 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.173420    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42d7t\" (UniqueName: \"kubernetes.io/projected/cbe17f7a-6da6-4b5a-bcc2-380ae08485b0-kube-api-access-42d7t\") pod \"node-exporter-fzt68\" (UID: \"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0\") " pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.260330 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:37.260266    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fzt68"
Jul 01 20:47:37.272331 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:47:37.272308    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe17f7a_6da6_4b5a_bcc2_380ae08485b0.slice/crio-b766cd2945ced16a82e2c7075d01934eab0f2c3517d1b42e8e28610a3c2942e2 WatchSource:0}: Error finding container b766cd2945ced16a82e2c7075d01934eab0f2c3517d1b42e8e28610a3c2942e2: Status 404 returned error can't find the container with id b766cd2945ced16a82e2c7075d01934eab0f2c3517d1b42e8e28610a3c2942e2
Jul 01 20:47:38.197230 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:38.197168    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fzt68" event={"ID":"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0","Type":"ContainerStarted","Data":"b766cd2945ced16a82e2c7075d01934eab0f2c3517d1b42e8e28610a3c2942e2"}
Jul 01 20:47:39.202294 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:39.202256    2573 generic.go:356] "Generic (PLEG): container finished" podID="cbe17f7a-6da6-4b5a-bcc2-380ae08485b0" containerID="b00a128c40544659745ed9c6a272770967a46357ada6bcbc598090fd549f645c" exitCode=0
Jul 01 20:47:39.202711 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:39.202348    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fzt68" event={"ID":"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0","Type":"ContainerDied","Data":"b00a128c40544659745ed9c6a272770967a46357ada6bcbc598090fd549f645c"}
Jul 01 20:47:40.209020 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:40.208988    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fzt68" event={"ID":"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0","Type":"ContainerStarted","Data":"5e55b3891c07ba3f61fc0065fefe5b9d3e432a6f5d7b32df3f1f2213d03b8577"}
Jul 01 20:47:40.209366 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:40.209037    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fzt68" event={"ID":"cbe17f7a-6da6-4b5a-bcc2-380ae08485b0","Type":"ContainerStarted","Data":"840bb707b03143e03d7313e2488e20c535aed7bc4aedd0e6ac621be006aed461"}
Jul 01 20:47:40.233583 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:40.233537    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fzt68" podStartSLOduration=3.090152409 podStartE2EDuration="4.233521461s" totalImagesPullingTime="1.076247103s" totalInitContainerRuntime="67.121949ms" isStatefulPod=true podCreationTimestamp="2026-07-01 20:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:40.232456373 +0000 UTC m=+93.143790999" watchObservedRunningTime="2026-07-01 20:47:40.233521461 +0000 UTC m=+93.144856087"
Jul 01 20:47:40.586276 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:40.586252    2573 scope.go:117] "RemoveContainer" containerID="8d51d822d4d84dd1b6ca4988a27aa59fc63cf82f3b0e4ed27ca3b33d7b81211a"
Jul 01 20:47:41.219683 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:41.219655    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 20:47:41.220156 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:41.219780    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" event={"ID":"4861b515-af80-46a0-a870-438254fa8c38","Type":"ContainerStarted","Data":"6a1abfb53a800ac0b9c0f7d4f612fe6cdc78da6de18dea016502aea166b583d7"}
Jul 01 20:47:41.220358 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:41.220331    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:47:41.239804 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:41.239758    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x" podStartSLOduration=50.90550337 podStartE2EDuration="58.239743175s" totalImagesPullingTime="7.334239805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:41.238606424 +0000 UTC m=+94.149941052" watchObservedRunningTime="2026-07-01 20:47:41.239743175 +0000 UTC m=+94.151077801"
Jul 01 20:47:41.557271 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:41.557198    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-r5d2x"
Jul 01 20:47:43.149873 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:43.149830    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74756dfdf5-52bcr"]
Jul 01 20:47:46.063266 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:46.063234    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wzfcf"
Jul 01 20:47:58.291674 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:58.291640    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc" containerID="57a4dd4a1bc7ddd15b4669f573e46121535f11ea59bb5cb0b4728e7325ad0e7c" exitCode=0
Jul 01 20:47:58.292067 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:58.291713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv" event={"ID":"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc","Type":"ContainerDied","Data":"57a4dd4a1bc7ddd15b4669f573e46121535f11ea59bb5cb0b4728e7325ad0e7c"}
Jul 01 20:47:58.292067 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:58.292056    2573 scope.go:117] "RemoveContainer" containerID="57a4dd4a1bc7ddd15b4669f573e46121535f11ea59bb5cb0b4728e7325ad0e7c"
Jul 01 20:47:59.297615 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:47:59.297579    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-6j4dv" event={"ID":"e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc","Type":"ContainerStarted","Data":"1d5608fd1d4591cae5cb6a506cab892133aff94da7e7692fbbdd32bc40bdc1c4"}
Jul 01 20:48:08.170346 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.170301    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr" podUID="08e4dcd0-b1bf-4ec9-a379-68343eec3f23" containerName="registry" containerID="cri-o://8dc3dc2ff036114f2ce06fed2e29cf9f5577b6aba7da39d1ad0070f3f067d8ef" gracePeriod=30
Jul 01 20:48:08.339342 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.339311    2573 generic.go:356] "Generic (PLEG): container finished" podID="08e4dcd0-b1bf-4ec9-a379-68343eec3f23" containerID="8dc3dc2ff036114f2ce06fed2e29cf9f5577b6aba7da39d1ad0070f3f067d8ef" exitCode=0
Jul 01 20:48:08.339447 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.339381    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr" event={"ID":"08e4dcd0-b1bf-4ec9-a379-68343eec3f23","Type":"ContainerDied","Data":"8dc3dc2ff036114f2ce06fed2e29cf9f5577b6aba7da39d1ad0070f3f067d8ef"}
Jul 01 20:48:08.341070 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.341052    2573 generic.go:356] "Generic (PLEG): container finished" podID="97bad7a1-1048-413b-a19c-4768dfca09e5" containerID="a22c0e7232b2ff011910b2ab1a60b827358a504631e588c931e49ff6b8a47546" exitCode=0
Jul 01 20:48:08.341156 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.341087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-zs75b" event={"ID":"97bad7a1-1048-413b-a19c-4768dfca09e5","Type":"ContainerDied","Data":"a22c0e7232b2ff011910b2ab1a60b827358a504631e588c931e49ff6b8a47546"}
Jul 01 20:48:08.341402 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.341390    2573 scope.go:117] "RemoveContainer" containerID="a22c0e7232b2ff011910b2ab1a60b827358a504631e588c931e49ff6b8a47546"
Jul 01 20:48:08.424311 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.424255    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:48:08.518058 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518030    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-installation-pull-secrets\") pod \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") "
Jul 01 20:48:08.518213 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518077    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-trusted-ca\") pod \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") "
Jul 01 20:48:08.518213 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518095    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") pod \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") "
Jul 01 20:48:08.518213 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518133    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-image-registry-private-configuration\") pod \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") "
Jul 01 20:48:08.518369 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518270    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvkpz\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-kube-api-access-bvkpz\") pod \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") "
Jul 01 20:48:08.518369 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518354    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-ca-trust-extracted\") pod \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") "
Jul 01 20:48:08.518467 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518391    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-certificates\") pod \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") "
Jul 01 20:48:08.518467 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518421    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-bound-sa-token\") pod \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\" (UID: \"08e4dcd0-b1bf-4ec9-a379-68343eec3f23\") "
Jul 01 20:48:08.518562 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518489    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "08e4dcd0-b1bf-4ec9-a379-68343eec3f23" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:08.518751 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.518726    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-trusted-ca\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:48:08.519083 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.519059    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "08e4dcd0-b1bf-4ec9-a379-68343eec3f23" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:48:08.520909 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.520844    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-kube-api-access-bvkpz" (OuterVolumeSpecName: "kube-api-access-bvkpz") pod "08e4dcd0-b1bf-4ec9-a379-68343eec3f23" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23"). InnerVolumeSpecName "kube-api-access-bvkpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:48:08.521145 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.521115    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "08e4dcd0-b1bf-4ec9-a379-68343eec3f23" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:48:08.521254 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.521140    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "08e4dcd0-b1bf-4ec9-a379-68343eec3f23" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:48:08.521324 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.521295    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "08e4dcd0-b1bf-4ec9-a379-68343eec3f23" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:48:08.521360 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.521313    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "08e4dcd0-b1bf-4ec9-a379-68343eec3f23" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:48:08.528700 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.528670    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "08e4dcd0-b1bf-4ec9-a379-68343eec3f23" (UID: "08e4dcd0-b1bf-4ec9-a379-68343eec3f23"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:48:08.619995 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.619971    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-installation-pull-secrets\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:48:08.619995 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.619993    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-tls\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:48:08.620150 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.620004    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-image-registry-private-configuration\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:48:08.620150 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.620014    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvkpz\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-kube-api-access-bvkpz\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:48:08.620150 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.620023    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-ca-trust-extracted\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:48:08.620150 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.620031    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-registry-certificates\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:48:08.620150 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:08.620039    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08e4dcd0-b1bf-4ec9-a379-68343eec3f23-bound-sa-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:48:09.347035 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:09.346995    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-zs75b" event={"ID":"97bad7a1-1048-413b-a19c-4768dfca09e5","Type":"ContainerStarted","Data":"c1c0d6e197d4899d62c0168210a0b9f0fca6f707adf631421b625f83cf4a8bcb"}
Jul 01 20:48:09.348237 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:09.348218    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr"
Jul 01 20:48:09.348237 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:09.348223    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74756dfdf5-52bcr" event={"ID":"08e4dcd0-b1bf-4ec9-a379-68343eec3f23","Type":"ContainerDied","Data":"7bbf1852aec5706e73e05786398dd0f0b48d10141767f07e205fd8580cb5351d"}
Jul 01 20:48:09.348372 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:09.348269    2573 scope.go:117] "RemoveContainer" containerID="8dc3dc2ff036114f2ce06fed2e29cf9f5577b6aba7da39d1ad0070f3f067d8ef"
Jul 01 20:48:09.382291 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:09.382266    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74756dfdf5-52bcr"]
Jul 01 20:48:09.387576 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:09.387552    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74756dfdf5-52bcr"]
Jul 01 20:48:09.591006 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:09.590971    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08e4dcd0-b1bf-4ec9-a379-68343eec3f23" path="/var/lib/kubelet/pods/08e4dcd0-b1bf-4ec9-a379-68343eec3f23/volumes"
Jul 01 20:48:23.417380 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:23.417347    2573 generic.go:356] "Generic (PLEG): container finished" podID="a812588d-2c2a-4559-89f2-c8ade3dec205" containerID="b6c7177d89c8ffee2df145ec01cd7f66942990b176536d0dff7675d1435f6df2" exitCode=0
Jul 01 20:48:23.417767 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:23.417422    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7" event={"ID":"a812588d-2c2a-4559-89f2-c8ade3dec205","Type":"ContainerDied","Data":"b6c7177d89c8ffee2df145ec01cd7f66942990b176536d0dff7675d1435f6df2"}
Jul 01 20:48:23.417767 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:23.417743    2573 scope.go:117] "RemoveContainer" containerID="b6c7177d89c8ffee2df145ec01cd7f66942990b176536d0dff7675d1435f6df2"
Jul 01 20:48:24.423243 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:48:24.423210    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-rfqh7" event={"ID":"a812588d-2c2a-4559-89f2-c8ade3dec205","Type":"ContainerStarted","Data":"4559cde764def2660bf073cc915e66c4f2b2e4b0e86a1813fbb1e79e276fd31c"}
Jul 01 20:51:07.496655 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:51:07.496626    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 20:51:07.503776 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:51:07.503751    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 20:51:07.507080 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:51:07.507054    2573 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 20:51:07.507973 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:51:07.507956    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 20:51:07.515345 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:51:07.515329    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 20:54:30.021032 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.020997    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"]
Jul 01 20:54:30.021461 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.021302    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08e4dcd0-b1bf-4ec9-a379-68343eec3f23" containerName="registry"
Jul 01 20:54:30.021461 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.021313    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="08e4dcd0-b1bf-4ec9-a379-68343eec3f23" containerName="registry"
Jul 01 20:54:30.021461 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.021386    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="08e4dcd0-b1bf-4ec9-a379-68343eec3f23" containerName="registry"
Jul 01 20:54:30.024293 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.024277    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.027656 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.027634    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 01 20:54:30.027841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.027829    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 01 20:54:30.029044 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.029023    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 01 20:54:30.029142 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.029112    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:54:30.029198 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.029111    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dkqv8\""
Jul 01 20:54:30.029198 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.029171    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 01 20:54:30.035191 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.035171    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"]
Jul 01 20:54:30.175422 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.175389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hfx\" (UniqueName: \"kubernetes.io/projected/f9889cdb-b6e8-433c-96fb-e67296d1279b-kube-api-access-c2hfx\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.175558 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.175430    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9889cdb-b6e8-433c-96fb-e67296d1279b-manager-config\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.175558 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.175455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9889cdb-b6e8-433c-96fb-e67296d1279b-cert\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.175558 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.175544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9889cdb-b6e8-433c-96fb-e67296d1279b-metrics-cert\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.276528 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.276471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hfx\" (UniqueName: \"kubernetes.io/projected/f9889cdb-b6e8-433c-96fb-e67296d1279b-kube-api-access-c2hfx\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.276528 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.276508    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9889cdb-b6e8-433c-96fb-e67296d1279b-manager-config\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.276642 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.276543    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9889cdb-b6e8-433c-96fb-e67296d1279b-cert\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.276642 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.276593    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9889cdb-b6e8-433c-96fb-e67296d1279b-metrics-cert\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.277155 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.277132    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9889cdb-b6e8-433c-96fb-e67296d1279b-manager-config\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.279051 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.279032    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9889cdb-b6e8-433c-96fb-e67296d1279b-metrics-cert\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.279149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.279086    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9889cdb-b6e8-433c-96fb-e67296d1279b-cert\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.285523 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.285501    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hfx\" (UniqueName: \"kubernetes.io/projected/f9889cdb-b6e8-433c-96fb-e67296d1279b-kube-api-access-c2hfx\") pod \"lws-controller-manager-55f5fcc78b-k2lxv\" (UID: \"f9889cdb-b6e8-433c-96fb-e67296d1279b\") " pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.333465 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.333433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:30.460156 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.460135    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"]
Jul 01 20:54:30.462578 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:54:30.462555    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9889cdb_b6e8_433c_96fb_e67296d1279b.slice/crio-b2e83986c4a3395724198e872f4e2894760dda297c184b7b42923f770f8ed942 WatchSource:0}: Error finding container b2e83986c4a3395724198e872f4e2894760dda297c184b7b42923f770f8ed942: Status 404 returned error can't find the container with id b2e83986c4a3395724198e872f4e2894760dda297c184b7b42923f770f8ed942
Jul 01 20:54:30.464654 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.464638    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:54:30.835139 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:30.835097    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv" event={"ID":"f9889cdb-b6e8-433c-96fb-e67296d1279b","Type":"ContainerStarted","Data":"b2e83986c4a3395724198e872f4e2894760dda297c184b7b42923f770f8ed942"}
Jul 01 20:54:34.855197 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:34.855160    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv" event={"ID":"f9889cdb-b6e8-433c-96fb-e67296d1279b","Type":"ContainerStarted","Data":"7f7a5449ae6954f9e8148358da90f6d464fc4ac3980b016ae5685261dcee4a51"}
Jul 01 20:54:34.855589 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:34.855292    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:34.874190 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:34.874144    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv" podStartSLOduration=2.456123257 podStartE2EDuration="5.874128273s" totalImagesPullingTime="3.418005016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:54:34.872228856 +0000 UTC m=+507.783563484" watchObservedRunningTime="2026-07-01 20:54:34.874128273 +0000 UTC m=+507.785462898"
Jul 01 20:54:45.861989 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:45.861961    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-55f5fcc78b-k2lxv"
Jul 01 20:54:49.296324 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.296291    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"]
Jul 01 20:54:49.302283 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.302264    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"
Jul 01 20:54:49.305887 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.305848    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 01 20:54:49.306058 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.306040    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zzrfz\""
Jul 01 20:54:49.307073 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.307057    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 01 20:54:49.312382 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.312360    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"]
Jul 01 20:54:49.312664 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.312649    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcp9w\" (UniqueName: \"kubernetes.io/projected/4de5123a-602a-438b-8683-4b39c6cf2c62-kube-api-access-hcp9w\") pod \"limitador-operator-controller-manager-6f9f468797-f448h\" (UID: \"4de5123a-602a-438b-8683-4b39c6cf2c62\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"
Jul 01 20:54:49.414061 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.414027    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcp9w\" (UniqueName: \"kubernetes.io/projected/4de5123a-602a-438b-8683-4b39c6cf2c62-kube-api-access-hcp9w\") pod \"limitador-operator-controller-manager-6f9f468797-f448h\" (UID: \"4de5123a-602a-438b-8683-4b39c6cf2c62\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"
Jul 01 20:54:49.423831 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.423801    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcp9w\" (UniqueName: \"kubernetes.io/projected/4de5123a-602a-438b-8683-4b39c6cf2c62-kube-api-access-hcp9w\") pod \"limitador-operator-controller-manager-6f9f468797-f448h\" (UID: \"4de5123a-602a-438b-8683-4b39c6cf2c62\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"
Jul 01 20:54:49.612761 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.612735    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"
Jul 01 20:54:49.742267 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.742232    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"]
Jul 01 20:54:49.745744 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:54:49.745714    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de5123a_602a_438b_8683_4b39c6cf2c62.slice/crio-c24d288e6ad4f4ad417597569c2faf3029ffbb407dd5afd0011400428f242f5d WatchSource:0}: Error finding container c24d288e6ad4f4ad417597569c2faf3029ffbb407dd5afd0011400428f242f5d: Status 404 returned error can't find the container with id c24d288e6ad4f4ad417597569c2faf3029ffbb407dd5afd0011400428f242f5d
Jul 01 20:54:49.917465 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:49.917385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h" event={"ID":"4de5123a-602a-438b-8683-4b39c6cf2c62","Type":"ContainerStarted","Data":"c24d288e6ad4f4ad417597569c2faf3029ffbb407dd5afd0011400428f242f5d"}
Jul 01 20:54:51.928313 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:51.928275    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h" event={"ID":"4de5123a-602a-438b-8683-4b39c6cf2c62","Type":"ContainerStarted","Data":"5028e0c54ba05bf7df38304315220dc5433b36b7da3d01ebb4109a1a3688df9b"}
Jul 01 20:54:51.928683 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:51.928392    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"
Jul 01 20:54:51.950798 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:54:51.950748    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h" podStartSLOduration=0.915273088 podStartE2EDuration="2.950734278s" totalImagesPullingTime="2.03546119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:54:51.949476425 +0000 UTC m=+524.860811091" watchObservedRunningTime="2026-07-01 20:54:51.950734278 +0000 UTC m=+524.862068905"
Jul 01 20:55:02.934539 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:02.934464    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-f448h"
Jul 01 20:55:37.095175 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:37.095145    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-nw9cq"]
Jul 01 20:55:37.097370 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:37.097353    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-nw9cq"
Jul 01 20:55:37.100394 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:37.100372    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zkbf2\""
Jul 01 20:55:37.107481 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:37.107461    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-nw9cq"]
Jul 01 20:55:37.174811 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:37.174784    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czw58\" (UniqueName: \"kubernetes.io/projected/01be4aaa-4f6c-4456-9188-05c424cb0b7e-kube-api-access-czw58\") pod \"authorino-b6c89bb85-nw9cq\" (UID: \"01be4aaa-4f6c-4456-9188-05c424cb0b7e\") " pod="kuadrant-system/authorino-b6c89bb85-nw9cq"
Jul 01 20:55:37.275360 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:37.275328    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czw58\" (UniqueName: \"kubernetes.io/projected/01be4aaa-4f6c-4456-9188-05c424cb0b7e-kube-api-access-czw58\") pod \"authorino-b6c89bb85-nw9cq\" (UID: \"01be4aaa-4f6c-4456-9188-05c424cb0b7e\") " pod="kuadrant-system/authorino-b6c89bb85-nw9cq"
Jul 01 20:55:37.288339 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:37.288312    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czw58\" (UniqueName: \"kubernetes.io/projected/01be4aaa-4f6c-4456-9188-05c424cb0b7e-kube-api-access-czw58\") pod \"authorino-b6c89bb85-nw9cq\" (UID: \"01be4aaa-4f6c-4456-9188-05c424cb0b7e\") " pod="kuadrant-system/authorino-b6c89bb85-nw9cq"
Jul 01 20:55:37.406895 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:37.406810    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-nw9cq"
Jul 01 20:55:37.532904 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:37.532741    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-nw9cq"]
Jul 01 20:55:37.535752 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:55:37.535724    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01be4aaa_4f6c_4456_9188_05c424cb0b7e.slice/crio-6897eda45315d3e93d50140b535d960d1e4a24b4f79b52dc1ec0c7f721312d8e WatchSource:0}: Error finding container 6897eda45315d3e93d50140b535d960d1e4a24b4f79b52dc1ec0c7f721312d8e: Status 404 returned error can't find the container with id 6897eda45315d3e93d50140b535d960d1e4a24b4f79b52dc1ec0c7f721312d8e
Jul 01 20:55:38.133035 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:38.133001    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-nw9cq" event={"ID":"01be4aaa-4f6c-4456-9188-05c424cb0b7e","Type":"ContainerStarted","Data":"6897eda45315d3e93d50140b535d960d1e4a24b4f79b52dc1ec0c7f721312d8e"}
Jul 01 20:55:41.147202 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:41.147167    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-nw9cq" event={"ID":"01be4aaa-4f6c-4456-9188-05c424cb0b7e","Type":"ContainerStarted","Data":"45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1"}
Jul 01 20:55:41.166376 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:41.166333    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-nw9cq" podStartSLOduration=1.426705304 podStartE2EDuration="4.166318689s" totalImagesPullingTime="2.739613385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:55:41.164544706 +0000 UTC m=+574.075879331" watchObservedRunningTime="2026-07-01 20:55:41.166318689 +0000 UTC m=+574.077653315"
Jul 01 20:55:42.709827 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:42.709789    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-nw9cq"]
Jul 01 20:55:43.155360 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:43.155315    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-nw9cq" podUID="01be4aaa-4f6c-4456-9188-05c424cb0b7e" containerName="authorino" containerID="cri-o://45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1" gracePeriod=30
Jul 01 20:55:43.385699 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:43.385677    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-nw9cq"
Jul 01 20:55:43.523725 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:43.523649    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czw58\" (UniqueName: \"kubernetes.io/projected/01be4aaa-4f6c-4456-9188-05c424cb0b7e-kube-api-access-czw58\") pod \"01be4aaa-4f6c-4456-9188-05c424cb0b7e\" (UID: \"01be4aaa-4f6c-4456-9188-05c424cb0b7e\") "
Jul 01 20:55:43.525786 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:43.525765    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01be4aaa-4f6c-4456-9188-05c424cb0b7e-kube-api-access-czw58" (OuterVolumeSpecName: "kube-api-access-czw58") pod "01be4aaa-4f6c-4456-9188-05c424cb0b7e" (UID: "01be4aaa-4f6c-4456-9188-05c424cb0b7e"). InnerVolumeSpecName "kube-api-access-czw58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:55:43.625137 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:43.625110    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czw58\" (UniqueName: \"kubernetes.io/projected/01be4aaa-4f6c-4456-9188-05c424cb0b7e-kube-api-access-czw58\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:55:44.166274 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:44.166241    2573 generic.go:356] "Generic (PLEG): container finished" podID="01be4aaa-4f6c-4456-9188-05c424cb0b7e" containerID="45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1" exitCode=0
Jul 01 20:55:44.166676 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:44.166308    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-nw9cq"
Jul 01 20:55:44.166676 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:44.166330    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-nw9cq" event={"ID":"01be4aaa-4f6c-4456-9188-05c424cb0b7e","Type":"ContainerDied","Data":"45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1"}
Jul 01 20:55:44.166676 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:44.166375    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-nw9cq" event={"ID":"01be4aaa-4f6c-4456-9188-05c424cb0b7e","Type":"ContainerDied","Data":"6897eda45315d3e93d50140b535d960d1e4a24b4f79b52dc1ec0c7f721312d8e"}
Jul 01 20:55:44.166676 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:44.166393    2573 scope.go:117] "RemoveContainer" containerID="45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1"
Jul 01 20:55:44.174822 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:44.174800    2573 scope.go:117] "RemoveContainer" containerID="45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1"
Jul 01 20:55:44.175103 ip-10-0-138-219 kubenswrapper[2573]: E0701 20:55:44.175085    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1\": container with ID starting with 45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1 not found: ID does not exist" containerID="45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1"
Jul 01 20:55:44.175155 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:44.175109    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1"} err="failed to get container status \"45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1\": rpc error: code = NotFound desc = could not find container \"45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1\": container with ID starting with 45810fe64c398e4ad408d1cd114ea3a142f5ee12a0fe25027de5c3bc4a7f82a1 not found: ID does not exist"
Jul 01 20:55:44.182466 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:44.182444    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-nw9cq"]
Jul 01 20:55:44.186349 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:44.186328    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-nw9cq"]
Jul 01 20:55:45.591384 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:55:45.591356    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01be4aaa-4f6c-4456-9188-05c424cb0b7e" path="/var/lib/kubelet/pods/01be4aaa-4f6c-4456-9188-05c424cb0b7e/volumes"
Jul 01 20:56:07.530315 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:56:07.530288    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 20:56:07.533448 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:56:07.533425    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 20:56:07.536967 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:56:07.536948    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 20:56:07.539920 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:56:07.539903    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 20:57:00.863366 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.863330    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-zqr6x"]
Jul 01 20:57:00.865714 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.863649    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01be4aaa-4f6c-4456-9188-05c424cb0b7e" containerName="authorino"
Jul 01 20:57:00.865714 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.863662    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="01be4aaa-4f6c-4456-9188-05c424cb0b7e" containerName="authorino"
Jul 01 20:57:00.865714 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.863717    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="01be4aaa-4f6c-4456-9188-05c424cb0b7e" containerName="authorino"
Jul 01 20:57:00.866548 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.866531    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zqr6x"
Jul 01 20:57:00.870997 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.870967    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 20:57:00.870997 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.870990    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 20:57:00.871192 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.871008    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7ntwt\""
Jul 01 20:57:00.871192 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.871105    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 20:57:00.875483 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.875455    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zqr6x"]
Jul 01 20:57:00.995364 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.995338    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf65x\" (UniqueName: \"kubernetes.io/projected/e8442666-24ad-40ef-ac08-ced96e706f5a-kube-api-access-kf65x\") pod \"s3-init-zqr6x\" (UID: \"e8442666-24ad-40ef-ac08-ced96e706f5a\") " pod="kserve/s3-init-zqr6x"
Jul 01 20:57:00.995489 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:00.995393    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8442666-24ad-40ef-ac08-ced96e706f5a-model-cache\") pod \"s3-init-zqr6x\" (UID: \"e8442666-24ad-40ef-ac08-ced96e706f5a\") " pod="kserve/s3-init-zqr6x"
Jul 01 20:57:01.095800 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:01.095775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf65x\" (UniqueName: \"kubernetes.io/projected/e8442666-24ad-40ef-ac08-ced96e706f5a-kube-api-access-kf65x\") pod \"s3-init-zqr6x\" (UID: \"e8442666-24ad-40ef-ac08-ced96e706f5a\") " pod="kserve/s3-init-zqr6x"
Jul 01 20:57:01.095950 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:01.095829    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8442666-24ad-40ef-ac08-ced96e706f5a-model-cache\") pod \"s3-init-zqr6x\" (UID: \"e8442666-24ad-40ef-ac08-ced96e706f5a\") " pod="kserve/s3-init-zqr6x"
Jul 01 20:57:01.096194 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:01.096175    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8442666-24ad-40ef-ac08-ced96e706f5a-model-cache\") pod \"s3-init-zqr6x\" (UID: \"e8442666-24ad-40ef-ac08-ced96e706f5a\") " pod="kserve/s3-init-zqr6x"
Jul 01 20:57:01.105351 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:01.105325    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf65x\" (UniqueName: \"kubernetes.io/projected/e8442666-24ad-40ef-ac08-ced96e706f5a-kube-api-access-kf65x\") pod \"s3-init-zqr6x\" (UID: \"e8442666-24ad-40ef-ac08-ced96e706f5a\") " pod="kserve/s3-init-zqr6x"
Jul 01 20:57:01.178832 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:01.178773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zqr6x"
Jul 01 20:57:01.304944 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:01.304911    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zqr6x"]
Jul 01 20:57:01.307538 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:57:01.307513    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8442666_24ad_40ef_ac08_ced96e706f5a.slice/crio-eeb4464b916b51ae3a146e7b897f1fe0082c4d11f81f1f09623cd4e472b8fc93 WatchSource:0}: Error finding container eeb4464b916b51ae3a146e7b897f1fe0082c4d11f81f1f09623cd4e472b8fc93: Status 404 returned error can't find the container with id eeb4464b916b51ae3a146e7b897f1fe0082c4d11f81f1f09623cd4e472b8fc93
Jul 01 20:57:01.507911 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:01.507819    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zqr6x" event={"ID":"e8442666-24ad-40ef-ac08-ced96e706f5a","Type":"ContainerStarted","Data":"eeb4464b916b51ae3a146e7b897f1fe0082c4d11f81f1f09623cd4e472b8fc93"}
Jul 01 20:57:05.536755 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:05.536707    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zqr6x" event={"ID":"e8442666-24ad-40ef-ac08-ced96e706f5a","Type":"ContainerStarted","Data":"df527d7b3e351958c1fd5d0fcb962281b0f907f59ff1576a7da50221ee313174"}
Jul 01 20:57:07.546841 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:07.546811    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8442666-24ad-40ef-ac08-ced96e706f5a" containerID="df527d7b3e351958c1fd5d0fcb962281b0f907f59ff1576a7da50221ee313174" exitCode=0
Jul 01 20:57:07.547235 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:07.546883    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zqr6x" event={"ID":"e8442666-24ad-40ef-ac08-ced96e706f5a","Type":"ContainerDied","Data":"df527d7b3e351958c1fd5d0fcb962281b0f907f59ff1576a7da50221ee313174"}
Jul 01 20:57:11.955682 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:11.955659    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 20:57:12.575256 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:12.575222    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zqr6x" event={"ID":"e8442666-24ad-40ef-ac08-ced96e706f5a","Type":"ContainerStarted","Data":"666c6ef13f6f317fe76caa5111e04796f54fa77da9afa8f2aad251d8818a6ab1"}
Jul 01 20:57:12.593055 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:57:12.593001    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-zqr6x" podStartSLOduration=2.7916647489999997 podStartE2EDuration="12.592986225s" totalImagesPullingTime="8.027622682s" totalInitContainerRuntime="1.773698794s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:57:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:57:12.591465579 +0000 UTC m=+665.502800205" watchObservedRunningTime="2026-07-01 20:57:12.592986225 +0000 UTC m=+665.504320858"
Jul 01 20:58:16.848694 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:16.848659    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8442666-24ad-40ef-ac08-ced96e706f5a" containerID="666c6ef13f6f317fe76caa5111e04796f54fa77da9afa8f2aad251d8818a6ab1" exitCode=0
Jul 01 20:58:16.849135 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:16.848734    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zqr6x" event={"ID":"e8442666-24ad-40ef-ac08-ced96e706f5a","Type":"ContainerDied","Data":"666c6ef13f6f317fe76caa5111e04796f54fa77da9afa8f2aad251d8818a6ab1"}
Jul 01 20:58:17.979053 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:17.979034    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zqr6x"
Jul 01 20:58:18.094761 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:18.094737    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8442666-24ad-40ef-ac08-ced96e706f5a-model-cache\") pod \"e8442666-24ad-40ef-ac08-ced96e706f5a\" (UID: \"e8442666-24ad-40ef-ac08-ced96e706f5a\") "
Jul 01 20:58:18.094908 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:18.094779    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf65x\" (UniqueName: \"kubernetes.io/projected/e8442666-24ad-40ef-ac08-ced96e706f5a-kube-api-access-kf65x\") pod \"e8442666-24ad-40ef-ac08-ced96e706f5a\" (UID: \"e8442666-24ad-40ef-ac08-ced96e706f5a\") "
Jul 01 20:58:18.096900 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:18.096870    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8442666-24ad-40ef-ac08-ced96e706f5a-kube-api-access-kf65x" (OuterVolumeSpecName: "kube-api-access-kf65x") pod "e8442666-24ad-40ef-ac08-ced96e706f5a" (UID: "e8442666-24ad-40ef-ac08-ced96e706f5a"). InnerVolumeSpecName "kube-api-access-kf65x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:58:18.113390 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:18.113343    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8442666-24ad-40ef-ac08-ced96e706f5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8442666-24ad-40ef-ac08-ced96e706f5a" (UID: "e8442666-24ad-40ef-ac08-ced96e706f5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:58:18.196132 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:18.196108    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8442666-24ad-40ef-ac08-ced96e706f5a-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:58:18.196132 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:18.196129    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf65x\" (UniqueName: \"kubernetes.io/projected/e8442666-24ad-40ef-ac08-ced96e706f5a-kube-api-access-kf65x\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 20:58:18.858887 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:18.858842    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zqr6x"
Jul 01 20:58:18.858887 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:18.858879    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zqr6x" event={"ID":"e8442666-24ad-40ef-ac08-ced96e706f5a","Type":"ContainerDied","Data":"eeb4464b916b51ae3a146e7b897f1fe0082c4d11f81f1f09623cd4e472b8fc93"}
Jul 01 20:58:18.859086 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:18.858909    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb4464b916b51ae3a146e7b897f1fe0082c4d11f81f1f09623cd4e472b8fc93"
Jul 01 20:58:42.980726 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.980688    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"]
Jul 01 20:58:42.981250 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.981229    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8442666-24ad-40ef-ac08-ced96e706f5a" containerName="s3-init"
Jul 01 20:58:42.981294 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.981254    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8442666-24ad-40ef-ac08-ced96e706f5a" containerName="s3-init"
Jul 01 20:58:42.981294 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.981283    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8442666-24ad-40ef-ac08-ced96e706f5a" containerName="download-hf-model"
Jul 01 20:58:42.981294 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.981292    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8442666-24ad-40ef-ac08-ced96e706f5a" containerName="download-hf-model"
Jul 01 20:58:42.981392 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.981377    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8442666-24ad-40ef-ac08-ced96e706f5a" containerName="s3-init"
Jul 01 20:58:42.984048 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.984030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:42.987078 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.987053    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 20:58:42.987220 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.987104    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 20:58:42.987220 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.987104    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 20:58:42.987328 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.987217    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 01 20:58:42.988306 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.988289    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-76tz7\""
Jul 01 20:58:42.994433 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:42.994406    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"]
Jul 01 20:58:43.089026 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.088997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.089149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.089034    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.089149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.089092    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-home\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.089149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.089134    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zwx\" (UniqueName: \"kubernetes.io/projected/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kube-api-access-76zwx\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.089264 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.089162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tls-certs\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.089264 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.089181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-dshm\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.089264 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.089195    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-model-cache\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190002 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.189978    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190124 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190015    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190124 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190047    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-home\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190124 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190083    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zwx\" (UniqueName: \"kubernetes.io/projected/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kube-api-access-76zwx\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190124 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tls-certs\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190316 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-dshm\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190316 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190151    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-model-cache\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190486 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190463    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190557 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190496    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190557 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190542    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-home\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.190687 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.190574    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-model-cache\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.192564 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.192543    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-dshm\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.192700 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.192682    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tls-certs\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.199455 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.199432    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zwx\" (UniqueName: \"kubernetes.io/projected/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kube-api-access-76zwx\") pod \"precise-prefix-cache-test-kserve-7f56466b78-t94t8\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.295797 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.295737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:58:43.424528 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.424505    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"]
Jul 01 20:58:43.426623 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:58:43.426600    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3924e8_bb6b_4b0a_bd00_5d3ede90b060.slice/crio-eb810daa2a37128aa6d3b40898071d05a92f756efb5bd80a2cf3e452feb7c92f WatchSource:0}: Error finding container eb810daa2a37128aa6d3b40898071d05a92f756efb5bd80a2cf3e452feb7c92f: Status 404 returned error can't find the container with id eb810daa2a37128aa6d3b40898071d05a92f756efb5bd80a2cf3e452feb7c92f
Jul 01 20:58:43.534640 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.534601    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"]
Jul 01 20:58:43.537285 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.537266    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.540130 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.540108    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-cdn45\""
Jul 01 20:58:43.551912 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.551834    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"]
Jul 01 20:58:43.693779 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.693748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7674792c-d7fd-4157-87a7-08ae4702e7da-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.693976 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.693803    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.693976 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.693828    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.693976 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.693910    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.693976 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.693963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pht82\" (UniqueName: \"kubernetes.io/projected/7674792c-d7fd-4157-87a7-08ae4702e7da-kube-api-access-pht82\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.694149 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.693999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.794716 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.794678    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.794912 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.794734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.794912 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.794790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pht82\" (UniqueName: \"kubernetes.io/projected/7674792c-d7fd-4157-87a7-08ae4702e7da-kube-api-access-pht82\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.794912 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.794819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.795090 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.795067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7674792c-d7fd-4157-87a7-08ae4702e7da-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.795174 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.795147    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.795257 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.795185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.795320 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.795299    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.795423 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.795401    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.795482 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.795449    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.797798 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.797774    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7674792c-d7fd-4157-87a7-08ae4702e7da-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.803982 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.803930    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pht82\" (UniqueName: \"kubernetes.io/projected/7674792c-d7fd-4157-87a7-08ae4702e7da-kube-api-access-pht82\") pod \"precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.845940 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.845906    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:58:43.963470 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.963439    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8" event={"ID":"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060","Type":"ContainerStarted","Data":"5f60e700371227cb3e96a4f6d80a0d3f62100e0f2708ff9cecf11aa72fc98038"}
Jul 01 20:58:43.963610 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.963489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8" event={"ID":"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060","Type":"ContainerStarted","Data":"eb810daa2a37128aa6d3b40898071d05a92f756efb5bd80a2cf3e452feb7c92f"}
Jul 01 20:58:43.976386 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:43.976364    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"]
Jul 01 20:58:43.978723 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:58:43.978700    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7674792c_d7fd_4157_87a7_08ae4702e7da.slice/crio-0f06b193af572aae11c2316021b13fdb1832ba2183caeef6e6eb4c1b7292e8f8 WatchSource:0}: Error finding container 0f06b193af572aae11c2316021b13fdb1832ba2183caeef6e6eb4c1b7292e8f8: Status 404 returned error can't find the container with id 0f06b193af572aae11c2316021b13fdb1832ba2183caeef6e6eb4c1b7292e8f8
Jul 01 20:58:44.970642 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:44.970560    2573 generic.go:356] "Generic (PLEG): container finished" podID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerID="dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db" exitCode=0
Jul 01 20:58:44.971152 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:44.970646    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" event={"ID":"7674792c-d7fd-4157-87a7-08ae4702e7da","Type":"ContainerDied","Data":"dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db"}
Jul 01 20:58:44.971152 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:44.970709    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" event={"ID":"7674792c-d7fd-4157-87a7-08ae4702e7da","Type":"ContainerStarted","Data":"0f06b193af572aae11c2316021b13fdb1832ba2183caeef6e6eb4c1b7292e8f8"}
Jul 01 20:58:46.987061 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:46.987024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" event={"ID":"7674792c-d7fd-4157-87a7-08ae4702e7da","Type":"ContainerStarted","Data":"b8d265a3bfa6dcff615a92d68d15c6746a0d95e895a43ca9b598be65453864bd"}
Jul 01 20:58:57.041205 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:57.041090    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7_7674792c-d7fd-4157-87a7-08ae4702e7da/main/0.log"
Jul 01 20:58:57.041698 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:57.041567    2573 generic.go:356] "Generic (PLEG): container finished" podID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerID="b8d265a3bfa6dcff615a92d68d15c6746a0d95e895a43ca9b598be65453864bd" exitCode=1
Jul 01 20:58:57.041698 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:57.041632    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" event={"ID":"7674792c-d7fd-4157-87a7-08ae4702e7da","Type":"ContainerDied","Data":"b8d265a3bfa6dcff615a92d68d15c6746a0d95e895a43ca9b598be65453864bd"}
Jul 01 20:58:58.048210 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:58.048168    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" containerID="5f60e700371227cb3e96a4f6d80a0d3f62100e0f2708ff9cecf11aa72fc98038" exitCode=0
Jul 01 20:58:58.048610 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:58:58.048243    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8" event={"ID":"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060","Type":"ContainerDied","Data":"5f60e700371227cb3e96a4f6d80a0d3f62100e0f2708ff9cecf11aa72fc98038"}
Jul 01 20:59:00.063019 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:00.062906    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8" event={"ID":"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060","Type":"ContainerStarted","Data":"59fccd73ed3b5175ac7624da852dfc34065e63313c06e4774ee5846c3f1d6112"}
Jul 01 20:59:00.087234 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:00.087176    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8" podStartSLOduration=2.518466315 podStartE2EDuration="18.087155608s" totalImagesPullingTime="1.723736448s" totalInitContainerRuntime="13.844952845s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:59:00.084035666 +0000 UTC m=+772.995370293" watchObservedRunningTime="2026-07-01 20:59:00.087155608 +0000 UTC m=+772.998490236"
Jul 01 20:59:00.296008 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:00.295974    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:59:03.296296 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:03.296245    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:59:03.310960 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:03.310934    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:59:04.096444 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:04.096344    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 20:59:17.152006 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:17.151975    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7_7674792c-d7fd-4157-87a7-08ae4702e7da/main/0.log"
Jul 01 20:59:17.152517 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:17.152493    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" event={"ID":"7674792c-d7fd-4157-87a7-08ae4702e7da","Type":"ContainerStarted","Data":"3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf"}
Jul 01 20:59:17.152826 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:17.152809    2573 scope.go:117] "RemoveContainer" containerID="b8d265a3bfa6dcff615a92d68d15c6746a0d95e895a43ca9b598be65453864bd"
Jul 01 20:59:18.158995 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:18.158970    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7_7674792c-d7fd-4157-87a7-08ae4702e7da/main/0.log"
Jul 01 20:59:18.159380 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:18.159350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" event={"ID":"7674792c-d7fd-4157-87a7-08ae4702e7da","Type":"ContainerStarted","Data":"4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef"}
Jul 01 20:59:18.159512 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:18.159495    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:59:18.187400 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:18.187342    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" podStartSLOduration=3.173736806 podStartE2EDuration="35.187328251s" totalImagesPullingTime="31.357801757s" totalInitContainerRuntime="655.789688ms" isStatefulPod=false podCreationTimestamp="2026-07-01 20:58:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:59:18.181887548 +0000 UTC m=+791.093222173" watchObservedRunningTime="2026-07-01 20:59:18.187328251 +0000 UTC m=+791.098662877"
Jul 01 20:59:23.846726 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:23.846693    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:59:23.847291 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:23.846785    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:59:23.849103 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:23.849075    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:59:23.860018 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:59:23.859994    2573 logging.go:55] [core] [Channel #17 SubChannel #18]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.31:9003", ServerName: "10.132.0.31:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.31:9003: connect: connection refused"
Jul 01 20:59:24.194456 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:24.194379    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 20:59:24.847108 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:24.847069    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.31:9003\" within 1s: context deadline exceeded"
Jul 01 20:59:33.847072 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:59:33.846998    2573 logging.go:55] [core] [Channel #19 SubChannel #20]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.31:9003", ServerName: "10.132.0.31:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.31:9003: connect: connection refused"
Jul 01 20:59:34.847095 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:34.847029    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.31:9003\" within 1s: context deadline exceeded"
Jul 01 20:59:43.290987 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.290955    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"]
Jul 01 20:59:43.392733 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.392703    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"]
Jul 01 20:59:43.392921 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.392813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.396346 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.396323    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 01 20:59:43.413274 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.413249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-kserve-provision-location\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.413401 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.413284    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-tmp-dir\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.413401 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.413317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7ls\" (UniqueName: \"kubernetes.io/projected/dfff2e82-5aae-449e-86b8-35e4c3574670-kube-api-access-hn7ls\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.413401 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.413348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-dshm\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.413401 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.413380    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-model-cache\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.413621 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.413420    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfff2e82-5aae-449e-86b8-35e4c3574670-tls-certs\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.413621 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.413472    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-home\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.514555 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.514532    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-kserve-provision-location\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.514685 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.514563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-tmp-dir\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.514685 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.514587    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7ls\" (UniqueName: \"kubernetes.io/projected/dfff2e82-5aae-449e-86b8-35e4c3574670-kube-api-access-hn7ls\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.514685 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.514608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-dshm\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.514685 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.514636    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-model-cache\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.514685 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.514663    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfff2e82-5aae-449e-86b8-35e4c3574670-tls-certs\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.514954 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.514700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-home\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.515042 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.515020    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-tmp-dir\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.515093 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.515036    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-kserve-provision-location\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.515135 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.515083    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-home\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.515135 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.515119    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-model-cache\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.516933 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.516911    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-dshm\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.517351 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.517332    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfff2e82-5aae-449e-86b8-35e4c3574670-tls-certs\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.527618 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.527589    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7ls\" (UniqueName: \"kubernetes.io/projected/dfff2e82-5aae-449e-86b8-35e4c3574670-kube-api-access-hn7ls\") pod \"auth-enabled-test-kserve-d8474687d-xkkfx\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.703127 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.703099    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 20:59:43.872543 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:43.872517    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"]
Jul 01 20:59:43.875167 ip-10-0-138-219 kubenswrapper[2573]: W0701 20:59:43.875145    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfff2e82_5aae_449e_86b8_35e4c3574670.slice/crio-769401a317b068eb10780c959ef860e202b8781db8fe2573cdd1d0222e205d45 WatchSource:0}: Error finding container 769401a317b068eb10780c959ef860e202b8781db8fe2573cdd1d0222e205d45: Status 404 returned error can't find the container with id 769401a317b068eb10780c959ef860e202b8781db8fe2573cdd1d0222e205d45
Jul 01 20:59:44.285997 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:44.285919    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" event={"ID":"dfff2e82-5aae-449e-86b8-35e4c3574670","Type":"ContainerStarted","Data":"7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8"}
Jul 01 20:59:44.285997 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:44.285970    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" event={"ID":"dfff2e82-5aae-449e-86b8-35e4c3574670","Type":"ContainerStarted","Data":"769401a317b068eb10780c959ef860e202b8781db8fe2573cdd1d0222e205d45"}
Jul 01 20:59:54.333194 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:54.333160    2573 generic.go:356] "Generic (PLEG): container finished" podID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerID="7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8" exitCode=0
Jul 01 20:59:54.333586 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:54.333233    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" event={"ID":"dfff2e82-5aae-449e-86b8-35e4c3574670","Type":"ContainerDied","Data":"7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8"}
Jul 01 20:59:54.334480 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:54.334461    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:59:55.199789 ip-10-0-138-219 kubenswrapper[2573]: I0701 20:59:55.199757    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 21:00:00.265465 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:00.265127    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"]
Jul 01 21:00:00.273414 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:00.273015    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"]
Jul 01 21:00:00.274286 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:00.273677    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="tokenizer" containerID="cri-o://3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf" gracePeriod=60
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:00:00.286714    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef" cmd=["/bin/sleep","15"]
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:00.286775    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" message="" err=<
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:00:00.286791    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main" containerID="cri-o://4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef"
Jul 01 21:00:00.287190 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:00.286824    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main" containerID="cri-o://4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef" gracePeriod=60
Jul 01 21:00:01.376926 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.376892    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7_7674792c-d7fd-4157-87a7-08ae4702e7da/main/0.log"
Jul 01 21:00:01.377344 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.377273    2573 generic.go:356] "Generic (PLEG): container finished" podID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerID="4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef" exitCode=0
Jul 01 21:00:01.377344 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.377310    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" event={"ID":"7674792c-d7fd-4157-87a7-08ae4702e7da","Type":"ContainerDied","Data":"4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef"}
Jul 01 21:00:01.377439 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.377394    2573 scope.go:117] "RemoveContainer" containerID="b8d265a3bfa6dcff615a92d68d15c6746a0d95e895a43ca9b598be65453864bd"
Jul 01 21:00:01.844040 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.844016    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 21:00:01.976973 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.976890    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-tmp\") pod \"7674792c-d7fd-4157-87a7-08ae4702e7da\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") "
Jul 01 21:00:01.976973 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.976929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-cache\") pod \"7674792c-d7fd-4157-87a7-08ae4702e7da\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") "
Jul 01 21:00:01.976973 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.976965    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pht82\" (UniqueName: \"kubernetes.io/projected/7674792c-d7fd-4157-87a7-08ae4702e7da-kube-api-access-pht82\") pod \"7674792c-d7fd-4157-87a7-08ae4702e7da\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") "
Jul 01 21:00:01.977241 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977013    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-uds\") pod \"7674792c-d7fd-4157-87a7-08ae4702e7da\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") "
Jul 01 21:00:01.977241 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977039    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7674792c-d7fd-4157-87a7-08ae4702e7da-tls-certs\") pod \"7674792c-d7fd-4157-87a7-08ae4702e7da\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") "
Jul 01 21:00:01.977241 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977058    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-kserve-provision-location\") pod \"7674792c-d7fd-4157-87a7-08ae4702e7da\" (UID: \"7674792c-d7fd-4157-87a7-08ae4702e7da\") "
Jul 01 21:00:01.977392 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977319    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "7674792c-d7fd-4157-87a7-08ae4702e7da" (UID: "7674792c-d7fd-4157-87a7-08ae4702e7da"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:01.977392 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977344    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "7674792c-d7fd-4157-87a7-08ae4702e7da" (UID: "7674792c-d7fd-4157-87a7-08ae4702e7da"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:01.977392 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977353    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "7674792c-d7fd-4157-87a7-08ae4702e7da" (UID: "7674792c-d7fd-4157-87a7-08ae4702e7da"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:01.977501 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977425    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-tmp\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:01.977501 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977443    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:01.977501 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977455    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:01.977840 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.977816    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7674792c-d7fd-4157-87a7-08ae4702e7da" (UID: "7674792c-d7fd-4157-87a7-08ae4702e7da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:01.979231 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.979197    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7674792c-d7fd-4157-87a7-08ae4702e7da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7674792c-d7fd-4157-87a7-08ae4702e7da" (UID: "7674792c-d7fd-4157-87a7-08ae4702e7da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:00:01.979231 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:01.979209    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7674792c-d7fd-4157-87a7-08ae4702e7da-kube-api-access-pht82" (OuterVolumeSpecName: "kube-api-access-pht82") pod "7674792c-d7fd-4157-87a7-08ae4702e7da" (UID: "7674792c-d7fd-4157-87a7-08ae4702e7da"). InnerVolumeSpecName "kube-api-access-pht82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:00:02.078248 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.078203    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7674792c-d7fd-4157-87a7-08ae4702e7da-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:02.078248 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.078243    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7674792c-d7fd-4157-87a7-08ae4702e7da-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:02.078248 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.078254    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pht82\" (UniqueName: \"kubernetes.io/projected/7674792c-d7fd-4157-87a7-08ae4702e7da-kube-api-access-pht82\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:02.384493 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.384462    2573 generic.go:356] "Generic (PLEG): container finished" podID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerID="3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf" exitCode=0
Jul 01 21:00:02.384923 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.384496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" event={"ID":"7674792c-d7fd-4157-87a7-08ae4702e7da","Type":"ContainerDied","Data":"3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf"}
Jul 01 21:00:02.384923 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.384532    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"
Jul 01 21:00:02.384923 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.384567    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7" event={"ID":"7674792c-d7fd-4157-87a7-08ae4702e7da","Type":"ContainerDied","Data":"0f06b193af572aae11c2316021b13fdb1832ba2183caeef6e6eb4c1b7292e8f8"}
Jul 01 21:00:02.384923 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.384597    2573 scope.go:117] "RemoveContainer" containerID="4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef"
Jul 01 21:00:02.395027 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.394965    2573 scope.go:117] "RemoveContainer" containerID="3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf"
Jul 01 21:00:02.403445 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.403428    2573 scope.go:117] "RemoveContainer" containerID="dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db"
Jul 01 21:00:02.407547 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.407526    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"]
Jul 01 21:00:02.411339 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.411319    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7"]
Jul 01 21:00:02.411796 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.411782    2573 scope.go:117] "RemoveContainer" containerID="4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef"
Jul 01 21:00:02.412060 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:00:02.412040    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef\": container with ID starting with 4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef not found: ID does not exist" containerID="4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef"
Jul 01 21:00:02.412139 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.412072    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef"} err="failed to get container status \"4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef\": rpc error: code = NotFound desc = could not find container \"4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef\": container with ID starting with 4feec0c798a5cb0fe75d27073b10ca60a083e320f51bf8a657d21919963ff3ef not found: ID does not exist"
Jul 01 21:00:02.412139 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.412097    2573 scope.go:117] "RemoveContainer" containerID="3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf"
Jul 01 21:00:02.412357 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:00:02.412341    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf\": container with ID starting with 3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf not found: ID does not exist" containerID="3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf"
Jul 01 21:00:02.412397 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.412363    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf"} err="failed to get container status \"3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf\": rpc error: code = NotFound desc = could not find container \"3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf\": container with ID starting with 3dfd5f13bf847e8b8bddd6da54c15128c8e3fa28dd1f7d4b8d25bf2e560543cf not found: ID does not exist"
Jul 01 21:00:02.412397 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.412378    2573 scope.go:117] "RemoveContainer" containerID="dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db"
Jul 01 21:00:02.412607 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:00:02.412590    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db\": container with ID starting with dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db not found: ID does not exist" containerID="dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db"
Jul 01 21:00:02.412667 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:02.412610    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db"} err="failed to get container status \"dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db\": rpc error: code = NotFound desc = could not find container \"dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db\": container with ID starting with dba386e02ce7255f148ad4b51fab566632c1dd3f3dee8e4e8f2f47cec54097db not found: ID does not exist"
Jul 01 21:00:03.592905 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:03.592871    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" path="/var/lib/kubelet/pods/7674792c-d7fd-4157-87a7-08ae4702e7da/volumes"
Jul 01 21:00:15.298821 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:15.298760    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8" podUID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" containerName="main" containerID="cri-o://59fccd73ed3b5175ac7624da852dfc34065e63313c06e4774ee5846c3f1d6112" gracePeriod=45
Jul 01 21:00:15.452371 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:15.452337    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" containerID="59fccd73ed3b5175ac7624da852dfc34065e63313c06e4774ee5846c3f1d6112" exitCode=0
Jul 01 21:00:15.452498 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:15.452404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8" event={"ID":"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060","Type":"ContainerDied","Data":"59fccd73ed3b5175ac7624da852dfc34065e63313c06e4774ee5846c3f1d6112"}
Jul 01 21:00:16.119215 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.119184    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 21:00:16.212760 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.212673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tls-certs\") pod \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") "
Jul 01 21:00:16.212760 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.212719    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-dshm\") pod \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") "
Jul 01 21:00:16.213010 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.212769    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kserve-provision-location\") pod \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") "
Jul 01 21:00:16.213010 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.212803    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tmp-dir\") pod \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") "
Jul 01 21:00:16.213010 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.212880    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76zwx\" (UniqueName: \"kubernetes.io/projected/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kube-api-access-76zwx\") pod \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") "
Jul 01 21:00:16.213010 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.212904    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-model-cache\") pod \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") "
Jul 01 21:00:16.213010 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.212946    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-home\") pod \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\" (UID: \"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060\") "
Jul 01 21:00:16.213281 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.213136    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" (UID: "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:16.213281 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.213234    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:16.213370 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.213355    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-home" (OuterVolumeSpecName: "home") pod "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" (UID: "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:16.213428 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.213282    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" (UID: "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:16.214800 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.214775    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-dshm" (OuterVolumeSpecName: "dshm") pod "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" (UID: "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:16.214961 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.214836    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" (UID: "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:00:16.215096 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.215076    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kube-api-access-76zwx" (OuterVolumeSpecName: "kube-api-access-76zwx") pod "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" (UID: "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060"). InnerVolumeSpecName "kube-api-access-76zwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:00:16.266031 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.265990    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" (UID: "2e3924e8-bb6b-4b0a-bd00-5d3ede90b060"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:00:16.313983 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.313953    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:16.313983 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.313978    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:16.313983 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.313988    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:16.314416 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.314000    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76zwx\" (UniqueName: \"kubernetes.io/projected/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-kube-api-access-76zwx\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:16.314416 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.314013    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:16.314416 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.314022    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:00:16.460974 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.460937    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"
Jul 01 21:00:16.461134 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.460930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8" event={"ID":"2e3924e8-bb6b-4b0a-bd00-5d3ede90b060","Type":"ContainerDied","Data":"eb810daa2a37128aa6d3b40898071d05a92f756efb5bd80a2cf3e452feb7c92f"}
Jul 01 21:00:16.461134 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.461093    2573 scope.go:117] "RemoveContainer" containerID="59fccd73ed3b5175ac7624da852dfc34065e63313c06e4774ee5846c3f1d6112"
Jul 01 21:00:16.472086 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.472065    2573 scope.go:117] "RemoveContainer" containerID="5f60e700371227cb3e96a4f6d80a0d3f62100e0f2708ff9cecf11aa72fc98038"
Jul 01 21:00:16.486627 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.486603    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"]
Jul 01 21:00:16.490417 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:16.490395    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8"]
Jul 01 21:00:17.084478 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:17.084436    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8" podUID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 01 21:00:17.590593 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:17.590565    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" path="/var/lib/kubelet/pods/2e3924e8-bb6b-4b0a-bd00-5d3ede90b060/volumes"
Jul 01 21:00:31.734467 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.734427    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"]
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.734919    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" containerName="main"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.734938    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" containerName="main"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.734964    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.734972    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.734985    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="storage-initializer"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.734994    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="storage-initializer"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735006    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" containerName="storage-initializer"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735014    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" containerName="storage-initializer"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735023    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="tokenizer"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735032    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="tokenizer"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735046    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main"
Jul 01 21:00:31.735053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735053    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main"
Jul 01 21:00:31.735643 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735147    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="tokenizer"
Jul 01 21:00:31.735643 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735157    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main"
Jul 01 21:00:31.735643 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735166    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7674792c-d7fd-4157-87a7-08ae4702e7da" containerName="main"
Jul 01 21:00:31.735643 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.735177    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e3924e8-bb6b-4b0a-bd00-5d3ede90b060" containerName="main"
Jul 01 21:00:31.738282 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.738263    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:31.741371 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.741191    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-zcxvv\""
Jul 01 21:00:31.742369 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.742348    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 01 21:00:31.747000 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.746943    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"]
Jul 01 21:00:31.830655 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.830615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50057615-920d-4d0d-87a7-164423f0de0e-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:31.830819 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.830671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/50057615-920d-4d0d-87a7-164423f0de0e-kube-api-access-j4pkc\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:31.830819 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.830796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50057615-920d-4d0d-87a7-164423f0de0e-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:31.931755 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.931724    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50057615-920d-4d0d-87a7-164423f0de0e-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:31.931935 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.931767    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50057615-920d-4d0d-87a7-164423f0de0e-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:31.931935 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.931806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/50057615-920d-4d0d-87a7-164423f0de0e-kube-api-access-j4pkc\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:31.932180 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.932157    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50057615-920d-4d0d-87a7-164423f0de0e-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:31.934542 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.934517    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50057615-920d-4d0d-87a7-164423f0de0e-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:31.940980 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:31.940953    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/50057615-920d-4d0d-87a7-164423f0de0e-kube-api-access-j4pkc\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:32.051306 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:32.051218    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:32.878480 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:32.878459    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"]
Jul 01 21:00:32.880686 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:00:32.880662    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50057615_920d_4d0d_87a7_164423f0de0e.slice/crio-df19ee75a3f9e48c2c6bc4d40559a3cacd9d35df7c3c857ea97152a499d5ece0 WatchSource:0}: Error finding container df19ee75a3f9e48c2c6bc4d40559a3cacd9d35df7c3c857ea97152a499d5ece0: Status 404 returned error can't find the container with id df19ee75a3f9e48c2c6bc4d40559a3cacd9d35df7c3c857ea97152a499d5ece0
Jul 01 21:00:33.554992 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:33.554955    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn" event={"ID":"50057615-920d-4d0d-87a7-164423f0de0e","Type":"ContainerStarted","Data":"d127463b553dedce8980773f0f7a81f79fcc4371a5620e1b675b6f026af057fc"}
Jul 01 21:00:33.555206 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:33.555182    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn" event={"ID":"50057615-920d-4d0d-87a7-164423f0de0e","Type":"ContainerStarted","Data":"df19ee75a3f9e48c2c6bc4d40559a3cacd9d35df7c3c857ea97152a499d5ece0"}
Jul 01 21:00:33.555346 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:33.555333    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:00:33.557120 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:33.557096    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" event={"ID":"dfff2e82-5aae-449e-86b8-35e4c3574670","Type":"ContainerStarted","Data":"e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee"}
Jul 01 21:00:33.573956 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:33.573913    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn" podStartSLOduration=2.573897881 podStartE2EDuration="2.573897881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:00:33.572139946 +0000 UTC m=+866.483474587" watchObservedRunningTime="2026-07-01 21:00:33.573897881 +0000 UTC m=+866.485232540"
Jul 01 21:00:33.594003 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:33.593957    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podStartSLOduration=2.291414209 podStartE2EDuration="50.593943193s" totalImagesPullingTime="38.539809196s" totalInitContainerRuntime="9.762719788s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:00:33.59267672 +0000 UTC m=+866.504011345" watchObservedRunningTime="2026-07-01 21:00:33.593943193 +0000 UTC m=+866.505277818"
Jul 01 21:00:33.703819 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:33.703787    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 21:00:33.704003 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:33.703865    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 21:00:33.705567 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:33.705534    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:00:43.704529 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:43.704482    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:00:53.704449 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:00:53.704409    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:01:03.704327 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:03.704241    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:01:05.568695 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:05.568668    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:01:07.559663 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:07.559634    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:01:07.560807 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:07.560786    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:01:07.568845 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:07.568825    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:01:07.569767 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:07.569747    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:01:13.703778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:13.703730    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:01:23.703781 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:23.703730    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:01:25.568513 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.568459    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn" podUID="50057615-920d-4d0d-87a7-164423f0de0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 21:01:25.592737 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.592704    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"]
Jul 01 21:01:25.601427 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:01:25.601396    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 21:01:25.601427 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:01:25.601427 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:01:25.601427 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="d127463b553dedce8980773f0f7a81f79fcc4371a5620e1b675b6f026af057fc" cmd=["/bin/sleep","15"]
Jul 01 21:01:25.601692 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.601446    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn" message="" err=<
Jul 01 21:01:25.601692 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:01:25.601692 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:01:25.601692 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 21:01:25.601692 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:01:25.601463    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 21:01:25.601692 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:01:25.601692 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:01:25.601692 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn" podUID="50057615-920d-4d0d-87a7-164423f0de0e" containerName="main" containerID="cri-o://d127463b553dedce8980773f0f7a81f79fcc4371a5620e1b675b6f026af057fc"
Jul 01 21:01:25.601692 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.601505    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn" podUID="50057615-920d-4d0d-87a7-164423f0de0e" containerName="main" containerID="cri-o://d127463b553dedce8980773f0f7a81f79fcc4371a5620e1b675b6f026af057fc" gracePeriod=60
Jul 01 21:01:25.802239 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.802207    2573 generic.go:356] "Generic (PLEG): container finished" podID="50057615-920d-4d0d-87a7-164423f0de0e" containerID="d127463b553dedce8980773f0f7a81f79fcc4371a5620e1b675b6f026af057fc" exitCode=0
Jul 01 21:01:25.802375 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.802273    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn" event={"ID":"50057615-920d-4d0d-87a7-164423f0de0e","Type":"ContainerDied","Data":"d127463b553dedce8980773f0f7a81f79fcc4371a5620e1b675b6f026af057fc"}
Jul 01 21:01:25.852251 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.852231    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:01:25.897102 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.897073    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/50057615-920d-4d0d-87a7-164423f0de0e-kube-api-access-j4pkc\") pod \"50057615-920d-4d0d-87a7-164423f0de0e\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") "
Jul 01 21:01:25.897236 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.897167    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50057615-920d-4d0d-87a7-164423f0de0e-tls-certs\") pod \"50057615-920d-4d0d-87a7-164423f0de0e\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") "
Jul 01 21:01:25.897236 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.897203    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50057615-920d-4d0d-87a7-164423f0de0e-tokenizer-uds\") pod \"50057615-920d-4d0d-87a7-164423f0de0e\" (UID: \"50057615-920d-4d0d-87a7-164423f0de0e\") "
Jul 01 21:01:25.897522 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.897498    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50057615-920d-4d0d-87a7-164423f0de0e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "50057615-920d-4d0d-87a7-164423f0de0e" (UID: "50057615-920d-4d0d-87a7-164423f0de0e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:01:25.899104 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.899076    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50057615-920d-4d0d-87a7-164423f0de0e-kube-api-access-j4pkc" (OuterVolumeSpecName: "kube-api-access-j4pkc") pod "50057615-920d-4d0d-87a7-164423f0de0e" (UID: "50057615-920d-4d0d-87a7-164423f0de0e"). InnerVolumeSpecName "kube-api-access-j4pkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:01:25.899179 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.899105    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50057615-920d-4d0d-87a7-164423f0de0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50057615-920d-4d0d-87a7-164423f0de0e" (UID: "50057615-920d-4d0d-87a7-164423f0de0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:01:25.997953 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.997917    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50057615-920d-4d0d-87a7-164423f0de0e-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:01:25.997953 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.997950    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50057615-920d-4d0d-87a7-164423f0de0e-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:01:25.997953 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:25.997959    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/50057615-920d-4d0d-87a7-164423f0de0e-kube-api-access-j4pkc\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:01:26.808472 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:26.808444    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"
Jul 01 21:01:26.808472 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:26.808464    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn" event={"ID":"50057615-920d-4d0d-87a7-164423f0de0e","Type":"ContainerDied","Data":"df19ee75a3f9e48c2c6bc4d40559a3cacd9d35df7c3c857ea97152a499d5ece0"}
Jul 01 21:01:26.808980 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:26.808526    2573 scope.go:117] "RemoveContainer" containerID="d127463b553dedce8980773f0f7a81f79fcc4371a5620e1b675b6f026af057fc"
Jul 01 21:01:26.829625 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:26.829595    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"]
Jul 01 21:01:26.835252 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:26.835231    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn"]
Jul 01 21:01:27.592143 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:27.592105    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50057615-920d-4d0d-87a7-164423f0de0e" path="/var/lib/kubelet/pods/50057615-920d-4d0d-87a7-164423f0de0e/volumes"
Jul 01 21:01:33.704473 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:33.704425    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:01:43.704465 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:43.704412    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:01:48.713774 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.713740    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"]
Jul 01 21:01:48.714153 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.714072    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50057615-920d-4d0d-87a7-164423f0de0e" containerName="main"
Jul 01 21:01:48.714153 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.714084    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="50057615-920d-4d0d-87a7-164423f0de0e" containerName="main"
Jul 01 21:01:48.714153 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.714137    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="50057615-920d-4d0d-87a7-164423f0de0e" containerName="main"
Jul 01 21:01:48.717002 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.716983    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:48.720993 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.720970    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 01 21:01:48.721119 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.720977    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-t2qmx\""
Jul 01 21:01:48.726027 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.726004    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"]
Jul 01 21:01:48.787059 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.787029    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:48.787196 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.787071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:48.787196 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.787139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42qwb\" (UniqueName: \"kubernetes.io/projected/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-kube-api-access-42qwb\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:48.888391 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.888359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:48.888495 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.888422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:48.888495 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.888464    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42qwb\" (UniqueName: \"kubernetes.io/projected/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-kube-api-access-42qwb\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:48.888780 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.888760    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:48.890673 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.890653    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:48.897888 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:48.897865    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42qwb\" (UniqueName: \"kubernetes.io/projected/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-kube-api-access-42qwb\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:49.028714 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:49.028625    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:49.166792 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:49.166726    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"]
Jul 01 21:01:49.168946 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:01:49.168922    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148e1ee0_b697_4edd_a645_cb60c5e7b4d5.slice/crio-24a9eb24b14b912e541062d65cc4c3285ebbd1965e3bef87da6b35a1e687ac98 WatchSource:0}: Error finding container 24a9eb24b14b912e541062d65cc4c3285ebbd1965e3bef87da6b35a1e687ac98: Status 404 returned error can't find the container with id 24a9eb24b14b912e541062d65cc4c3285ebbd1965e3bef87da6b35a1e687ac98
Jul 01 21:01:49.915062 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:49.915030    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj" event={"ID":"148e1ee0-b697-4edd-a645-cb60c5e7b4d5","Type":"ContainerStarted","Data":"4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b"}
Jul 01 21:01:49.915427 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:49.915083    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj" event={"ID":"148e1ee0-b697-4edd-a645-cb60c5e7b4d5","Type":"ContainerStarted","Data":"24a9eb24b14b912e541062d65cc4c3285ebbd1965e3bef87da6b35a1e687ac98"}
Jul 01 21:01:49.915427 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:49.915147    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:01:49.935041 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:49.934998    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj" podStartSLOduration=1.934985577 podStartE2EDuration="1.934985577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:01:49.933265799 +0000 UTC m=+942.844600426" watchObservedRunningTime="2026-07-01 21:01:49.934985577 +0000 UTC m=+942.846320204"
Jul 01 21:01:53.704476 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:01:53.704430    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:02:03.704166 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:03.704126    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:02:13.704465 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:13.704420    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 01 21:02:20.920893 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:20.920840    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:02:23.713943 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:23.713915    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 21:02:23.722374 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:23.722351    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 21:02:27.879008 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:27.878972    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"]
Jul 01 21:02:42.902151 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:42.902115    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main" containerID="cri-o://e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee" gracePeriod=45
Jul 01 21:02:43.641623 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.641600    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 21:02:43.738315 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.738231    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-kserve-provision-location\") pod \"dfff2e82-5aae-449e-86b8-35e4c3574670\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") "
Jul 01 21:02:43.738315 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.738281    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-tmp-dir\") pod \"dfff2e82-5aae-449e-86b8-35e4c3574670\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") "
Jul 01 21:02:43.738542 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.738322    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-dshm\") pod \"dfff2e82-5aae-449e-86b8-35e4c3574670\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") "
Jul 01 21:02:43.738542 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.738340    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-home\") pod \"dfff2e82-5aae-449e-86b8-35e4c3574670\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") "
Jul 01 21:02:43.738542 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.738361    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-model-cache\") pod \"dfff2e82-5aae-449e-86b8-35e4c3574670\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") "
Jul 01 21:02:43.738542 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.738395    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn7ls\" (UniqueName: \"kubernetes.io/projected/dfff2e82-5aae-449e-86b8-35e4c3574670-kube-api-access-hn7ls\") pod \"dfff2e82-5aae-449e-86b8-35e4c3574670\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") "
Jul 01 21:02:43.738542 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.738420    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfff2e82-5aae-449e-86b8-35e4c3574670-tls-certs\") pod \"dfff2e82-5aae-449e-86b8-35e4c3574670\" (UID: \"dfff2e82-5aae-449e-86b8-35e4c3574670\") "
Jul 01 21:02:43.738787 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.738654    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfff2e82-5aae-449e-86b8-35e4c3574670" (UID: "dfff2e82-5aae-449e-86b8-35e4c3574670"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:02:43.739005 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.738983    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:02:43.739121 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.739101    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-home" (OuterVolumeSpecName: "home") pod "dfff2e82-5aae-449e-86b8-35e4c3574670" (UID: "dfff2e82-5aae-449e-86b8-35e4c3574670"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:02:43.740620 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.740587    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfff2e82-5aae-449e-86b8-35e4c3574670-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfff2e82-5aae-449e-86b8-35e4c3574670" (UID: "dfff2e82-5aae-449e-86b8-35e4c3574670"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:02:43.740726 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.740664    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfff2e82-5aae-449e-86b8-35e4c3574670-kube-api-access-hn7ls" (OuterVolumeSpecName: "kube-api-access-hn7ls") pod "dfff2e82-5aae-449e-86b8-35e4c3574670" (UID: "dfff2e82-5aae-449e-86b8-35e4c3574670"). InnerVolumeSpecName "kube-api-access-hn7ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:02:43.740819 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.740802    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-dshm" (OuterVolumeSpecName: "dshm") pod "dfff2e82-5aae-449e-86b8-35e4c3574670" (UID: "dfff2e82-5aae-449e-86b8-35e4c3574670"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:02:43.751234 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.751205    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfff2e82-5aae-449e-86b8-35e4c3574670" (UID: "dfff2e82-5aae-449e-86b8-35e4c3574670"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:02:43.792843 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.792809    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dfff2e82-5aae-449e-86b8-35e4c3574670" (UID: "dfff2e82-5aae-449e-86b8-35e4c3574670"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:02:43.840052 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.840026    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:02:43.840052 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.840046    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn7ls\" (UniqueName: \"kubernetes.io/projected/dfff2e82-5aae-449e-86b8-35e4c3574670-kube-api-access-hn7ls\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:02:43.840169 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.840056    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfff2e82-5aae-449e-86b8-35e4c3574670-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:02:43.840169 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.840066    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:02:43.840169 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.840075    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:02:43.840169 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:43.840082    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfff2e82-5aae-449e-86b8-35e4c3574670-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:02:44.176268 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.176228    2573 generic.go:356] "Generic (PLEG): container finished" podID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerID="e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee" exitCode=0
Jul 01 21:02:44.176715 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.176283    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"
Jul 01 21:02:44.176715 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.176312    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" event={"ID":"dfff2e82-5aae-449e-86b8-35e4c3574670","Type":"ContainerDied","Data":"e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee"}
Jul 01 21:02:44.176715 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.176363    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx" event={"ID":"dfff2e82-5aae-449e-86b8-35e4c3574670","Type":"ContainerDied","Data":"769401a317b068eb10780c959ef860e202b8781db8fe2573cdd1d0222e205d45"}
Jul 01 21:02:44.176715 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.176386    2573 scope.go:117] "RemoveContainer" containerID="e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee"
Jul 01 21:02:44.185959 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.185939    2573 scope.go:117] "RemoveContainer" containerID="7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8"
Jul 01 21:02:44.195278 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.195254    2573 scope.go:117] "RemoveContainer" containerID="e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee"
Jul 01 21:02:44.195640 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:02:44.195616    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee\": container with ID starting with e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee not found: ID does not exist" containerID="e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee"
Jul 01 21:02:44.195725 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.195651    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee"} err="failed to get container status \"e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee\": rpc error: code = NotFound desc = could not find container \"e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee\": container with ID starting with e39e0740f867bb75a892864062b9181fd5f88cc6a9aad48ed16038278bb3f0ee not found: ID does not exist"
Jul 01 21:02:44.195725 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.195673    2573 scope.go:117] "RemoveContainer" containerID="7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8"
Jul 01 21:02:44.196194 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:02:44.196172    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8\": container with ID starting with 7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8 not found: ID does not exist" containerID="7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8"
Jul 01 21:02:44.196272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.196198    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8"} err="failed to get container status \"7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8\": rpc error: code = NotFound desc = could not find container \"7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8\": container with ID starting with 7c4149e7688cb8b5d91ba496b7901f048c87cdfd4f5dc35f268f706d0303cce8 not found: ID does not exist"
Jul 01 21:02:44.199501 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.199480    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"]
Jul 01 21:02:44.203683 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:44.203665    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx"]
Jul 01 21:02:45.590913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:02:45.590880    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" path="/var/lib/kubelet/pods/dfff2e82-5aae-449e-86b8-35e4c3574670/volumes"
Jul 01 21:03:19.997610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:19.997579    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"]
Jul 01 21:03:19.998063 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:19.997894    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main"
Jul 01 21:03:19.998063 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:19.997906    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main"
Jul 01 21:03:19.998063 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:19.997918    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="storage-initializer"
Jul 01 21:03:19.998063 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:19.997924    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="storage-initializer"
Jul 01 21:03:19.998063 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:19.997985    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfff2e82-5aae-449e-86b8-35e4c3574670" containerName="main"
Jul 01 21:03:20.000344 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.000327    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.003436 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.003411    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:03:20.003545 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.003417    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 01 21:03:20.011420 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.011397    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"]
Jul 01 21:03:20.117991 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.117957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tmp-dir\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.118130 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.117999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tls-certs\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.118130 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.118028    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.118130 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.118096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-home\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.118272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.118139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndp7w\" (UniqueName: \"kubernetes.io/projected/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kube-api-access-ndp7w\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.118272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.118166    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-dshm\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.118272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.118186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-model-cache\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219322 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219293    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tmp-dir\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219458 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219329    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tls-certs\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219458 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219458 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-home\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219458 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219403    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndp7w\" (UniqueName: \"kubernetes.io/projected/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kube-api-access-ndp7w\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219458 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-dshm\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219690 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-model-cache\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219783 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219759    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219842 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219806    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-model-cache\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219941 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219906    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tmp-dir\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.219941 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.219925    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-home\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.221799 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.221779    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-dshm\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.222076 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.222059    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tls-certs\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.229006 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.228982    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndp7w\" (UniqueName: \"kubernetes.io/projected/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kube-api-access-ndp7w\") pod \"auth-invalid-token-test-kserve-6f6986f4-vmpkj\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.307906 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.307828    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"]
Jul 01 21:03:20.310370 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.310345    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:20.317690 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.317670    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.323071 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.322879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-w9bgf\""
Jul 01 21:03:20.328481 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.328454    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"]
Jul 01 21:03:20.421472 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.421440    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37768643-fcba-4354-8f32-1af0a20265bc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.421605 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.421488    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.421605 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.421527    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjbg\" (UniqueName: \"kubernetes.io/projected/37768643-fcba-4354-8f32-1af0a20265bc-kube-api-access-jxjbg\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.421722 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.421652    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.452904 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.452886    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"]
Jul 01 21:03:20.455587 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:03:20.455564    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e029cf_6e76_48b1_aeb6_baac8aa2e876.slice/crio-4e60d2bedb55df5926421d2bfcb9930f2d84b8ee44c75d848cf7f80f260b5a5a WatchSource:0}: Error finding container 4e60d2bedb55df5926421d2bfcb9930f2d84b8ee44c75d848cf7f80f260b5a5a: Status 404 returned error can't find the container with id 4e60d2bedb55df5926421d2bfcb9930f2d84b8ee44c75d848cf7f80f260b5a5a
Jul 01 21:03:20.522396 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.522368    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37768643-fcba-4354-8f32-1af0a20265bc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.522501 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.522405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.522501 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.522436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjbg\" (UniqueName: \"kubernetes.io/projected/37768643-fcba-4354-8f32-1af0a20265bc-kube-api-access-jxjbg\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.522608 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.522513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.522757 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.522734    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.522873 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.522840    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.524581 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.524560    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37768643-fcba-4354-8f32-1af0a20265bc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.533677 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.533656    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjbg\" (UniqueName: \"kubernetes.io/projected/37768643-fcba-4354-8f32-1af0a20265bc-kube-api-access-jxjbg\") pod \"auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.641190 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.641167    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:20.774036 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:20.774005    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"]
Jul 01 21:03:20.775045 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:03:20.775014    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37768643_fcba_4354_8f32_1af0a20265bc.slice/crio-92a16307c86a28fc70d9529f86967af3ca214341dd28018ba40e65faa5fda4e5 WatchSource:0}: Error finding container 92a16307c86a28fc70d9529f86967af3ca214341dd28018ba40e65faa5fda4e5: Status 404 returned error can't find the container with id 92a16307c86a28fc70d9529f86967af3ca214341dd28018ba40e65faa5fda4e5
Jul 01 21:03:21.338870 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:21.338818    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" event={"ID":"37768643-fcba-4354-8f32-1af0a20265bc","Type":"ContainerStarted","Data":"3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62"}
Jul 01 21:03:21.339290 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:21.338902    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" event={"ID":"37768643-fcba-4354-8f32-1af0a20265bc","Type":"ContainerStarted","Data":"92a16307c86a28fc70d9529f86967af3ca214341dd28018ba40e65faa5fda4e5"}
Jul 01 21:03:21.341184 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:21.341088    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" event={"ID":"83e029cf-6e76-48b1-aeb6-baac8aa2e876","Type":"ContainerStarted","Data":"c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833"}
Jul 01 21:03:21.341184 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:21.341138    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" event={"ID":"83e029cf-6e76-48b1-aeb6-baac8aa2e876","Type":"ContainerStarted","Data":"4e60d2bedb55df5926421d2bfcb9930f2d84b8ee44c75d848cf7f80f260b5a5a"}
Jul 01 21:03:22.346720 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:22.346687    2573 generic.go:356] "Generic (PLEG): container finished" podID="37768643-fcba-4354-8f32-1af0a20265bc" containerID="3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62" exitCode=0
Jul 01 21:03:22.347225 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:22.346782    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" event={"ID":"37768643-fcba-4354-8f32-1af0a20265bc","Type":"ContainerDied","Data":"3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62"}
Jul 01 21:03:23.352692 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:23.352656    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" event={"ID":"37768643-fcba-4354-8f32-1af0a20265bc","Type":"ContainerStarted","Data":"8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4"}
Jul 01 21:03:23.353237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:23.352777    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:03:23.374037 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:23.373984    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" podStartSLOduration=2.689161488 podStartE2EDuration="3.373965995s" totalImagesPullingTime="0s" totalInitContainerRuntime="684.804507ms" isStatefulPod=false podCreationTimestamp="2026-07-01 21:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:03:23.372830414 +0000 UTC m=+1036.284165051" watchObservedRunningTime="2026-07-01 21:03:23.373965995 +0000 UTC m=+1036.285300622"
Jul 01 21:03:25.362867 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:25.362826    2573 generic.go:356] "Generic (PLEG): container finished" podID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerID="c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833" exitCode=0
Jul 01 21:03:25.363246 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:25.362899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" event={"ID":"83e029cf-6e76-48b1-aeb6-baac8aa2e876","Type":"ContainerDied","Data":"c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833"}
Jul 01 21:03:26.369190 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:26.369153    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" event={"ID":"83e029cf-6e76-48b1-aeb6-baac8aa2e876","Type":"ContainerStarted","Data":"59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5"}
Jul 01 21:03:26.396849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:26.396789    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podStartSLOduration=3.249147914 podStartE2EDuration="7.396768161s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.147620247s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:03:26.394499999 +0000 UTC m=+1039.305834626" watchObservedRunningTime="2026-07-01 21:03:26.396768161 +0000 UTC m=+1039.308102788"
Jul 01 21:03:27.310726 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:27.310692    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:30.310542 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:30.310493    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:03:30.312103 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:30.312072    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:03:40.310785 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:40.310738    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:03:49.623590 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.623559    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"]
Jul 01 21:03:49.632782 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:03:49.632748    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 21:03:49.632782 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:03:49.632782 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:03:49.632782 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b" cmd=["/bin/sleep","15"]
Jul 01 21:03:49.633077 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.632796    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj" message="" err=<
Jul 01 21:03:49.633077 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:03:49.633077 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:03:49.633077 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 21:03:49.633077 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:03:49.632814    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 21:03:49.633077 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:03:49.633077 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:03:49.633077 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj" podUID="148e1ee0-b697-4edd-a645-cb60c5e7b4d5" containerName="main" containerID="cri-o://4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b"
Jul 01 21:03:49.633077 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.632869    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj" podUID="148e1ee0-b697-4edd-a645-cb60c5e7b4d5" containerName="main" containerID="cri-o://4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b" gracePeriod=60
Jul 01 21:03:49.874532 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.874475    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:03:49.965195 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.965164    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42qwb\" (UniqueName: \"kubernetes.io/projected/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-kube-api-access-42qwb\") pod \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") "
Jul 01 21:03:49.965366 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.965226    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tokenizer-uds\") pod \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") "
Jul 01 21:03:49.965366 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.965295    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tls-certs\") pod \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\" (UID: \"148e1ee0-b697-4edd-a645-cb60c5e7b4d5\") "
Jul 01 21:03:49.965491 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.965424    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "148e1ee0-b697-4edd-a645-cb60c5e7b4d5" (UID: "148e1ee0-b697-4edd-a645-cb60c5e7b4d5"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:03:49.967216 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.967186    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-kube-api-access-42qwb" (OuterVolumeSpecName: "kube-api-access-42qwb") pod "148e1ee0-b697-4edd-a645-cb60c5e7b4d5" (UID: "148e1ee0-b697-4edd-a645-cb60c5e7b4d5"). InnerVolumeSpecName "kube-api-access-42qwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:03:49.967321 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:49.967231    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "148e1ee0-b697-4edd-a645-cb60c5e7b4d5" (UID: "148e1ee0-b697-4edd-a645-cb60c5e7b4d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:03:50.066467 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.066433    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:03:50.066467 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.066462    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:03:50.066467 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.066474    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42qwb\" (UniqueName: \"kubernetes.io/projected/148e1ee0-b697-4edd-a645-cb60c5e7b4d5-kube-api-access-42qwb\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:03:50.311528 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.311433    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:03:50.482834 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.482797    2573 generic.go:356] "Generic (PLEG): container finished" podID="148e1ee0-b697-4edd-a645-cb60c5e7b4d5" containerID="4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b" exitCode=0
Jul 01 21:03:50.483001 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.482837    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"
Jul 01 21:03:50.483001 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.482876    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj" event={"ID":"148e1ee0-b697-4edd-a645-cb60c5e7b4d5","Type":"ContainerDied","Data":"4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b"}
Jul 01 21:03:50.483001 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.482932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj" event={"ID":"148e1ee0-b697-4edd-a645-cb60c5e7b4d5","Type":"ContainerDied","Data":"24a9eb24b14b912e541062d65cc4c3285ebbd1965e3bef87da6b35a1e687ac98"}
Jul 01 21:03:50.483001 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.482960    2573 scope.go:117] "RemoveContainer" containerID="4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b"
Jul 01 21:03:50.492889 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.492866    2573 scope.go:117] "RemoveContainer" containerID="4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b"
Jul 01 21:03:50.493192 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:03:50.493171    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b\": container with ID starting with 4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b not found: ID does not exist" containerID="4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b"
Jul 01 21:03:50.493248 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.493200    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b"} err="failed to get container status \"4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b\": rpc error: code = NotFound desc = could not find container \"4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b\": container with ID starting with 4dcf7183ab56321daf96badd164fb0e9f061394e905be4684c7453257934015b not found: ID does not exist"
Jul 01 21:03:50.504543 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.504520    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"]
Jul 01 21:03:50.509617 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:50.509595    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj"]
Jul 01 21:03:51.592771 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:51.592730    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="148e1ee0-b697-4edd-a645-cb60c5e7b4d5" path="/var/lib/kubelet/pods/148e1ee0-b697-4edd-a645-cb60c5e7b4d5/volumes"
Jul 01 21:03:54.360024 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:03:54.359995    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:04:00.311329 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:04:00.311234    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:04:10.310932 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:04:10.310879    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:04:20.311233 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:04:20.311184    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:04:30.310708 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:04:30.310666    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:04:40.311358 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:04:40.311316    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:04:50.311116 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:04:50.311067    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:05:00.311130 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:00.311084    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:05:10.311250 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:10.311210    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:05:20.320574 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:20.320544    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:05:20.328131 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:20.328106    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:05:22.163884 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.163830    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"]
Jul 01 21:05:22.165471 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.165446    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"]
Jul 01 21:05:22.172662 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:05:22.172623    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 21:05:22.172662 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:05:22.172662 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:05:22.172662 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4" cmd=["/bin/sleep","15"]
Jul 01 21:05:22.172900 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.172674    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" message="" err=<
Jul 01 21:05:22.172900 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:05:22.172900 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:05:22.172900 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 21:05:22.172900 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:05:22.172685    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 21:05:22.172900 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:05:22.172900 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:05:22.172900 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" podUID="37768643-fcba-4354-8f32-1af0a20265bc" containerName="main" containerID="cri-o://8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4"
Jul 01 21:05:22.172900 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.172724    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" podUID="37768643-fcba-4354-8f32-1af0a20265bc" containerName="main" containerID="cri-o://8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4" gracePeriod=60
Jul 01 21:05:22.413566 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.413539    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:05:22.437332 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.437279    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-kserve-provision-location\") pod \"37768643-fcba-4354-8f32-1af0a20265bc\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") "
Jul 01 21:05:22.437411 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.437330    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-tokenizer-uds\") pod \"37768643-fcba-4354-8f32-1af0a20265bc\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") "
Jul 01 21:05:22.437452 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.437426    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37768643-fcba-4354-8f32-1af0a20265bc-tls-certs\") pod \"37768643-fcba-4354-8f32-1af0a20265bc\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") "
Jul 01 21:05:22.437494 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.437480    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxjbg\" (UniqueName: \"kubernetes.io/projected/37768643-fcba-4354-8f32-1af0a20265bc-kube-api-access-jxjbg\") pod \"37768643-fcba-4354-8f32-1af0a20265bc\" (UID: \"37768643-fcba-4354-8f32-1af0a20265bc\") "
Jul 01 21:05:22.437596 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.437575    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "37768643-fcba-4354-8f32-1af0a20265bc" (UID: "37768643-fcba-4354-8f32-1af0a20265bc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:05:22.437721 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.437706    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:22.438003 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.437980    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37768643-fcba-4354-8f32-1af0a20265bc" (UID: "37768643-fcba-4354-8f32-1af0a20265bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:05:22.439386 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.439339    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37768643-fcba-4354-8f32-1af0a20265bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37768643-fcba-4354-8f32-1af0a20265bc" (UID: "37768643-fcba-4354-8f32-1af0a20265bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:05:22.439560 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.439538    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37768643-fcba-4354-8f32-1af0a20265bc-kube-api-access-jxjbg" (OuterVolumeSpecName: "kube-api-access-jxjbg") pod "37768643-fcba-4354-8f32-1af0a20265bc" (UID: "37768643-fcba-4354-8f32-1af0a20265bc"). InnerVolumeSpecName "kube-api-access-jxjbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:05:22.538990 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.538962    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37768643-fcba-4354-8f32-1af0a20265bc-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:22.538990 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.538989    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxjbg\" (UniqueName: \"kubernetes.io/projected/37768643-fcba-4354-8f32-1af0a20265bc-kube-api-access-jxjbg\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:22.539204 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.539001    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37768643-fcba-4354-8f32-1af0a20265bc-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:22.902484 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.902456    2573 generic.go:356] "Generic (PLEG): container finished" podID="37768643-fcba-4354-8f32-1af0a20265bc" containerID="8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4" exitCode=0
Jul 01 21:05:22.902644 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.902500    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" event={"ID":"37768643-fcba-4354-8f32-1af0a20265bc","Type":"ContainerDied","Data":"8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4"}
Jul 01 21:05:22.902644 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.902506    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"
Jul 01 21:05:22.902644 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.902536    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts" event={"ID":"37768643-fcba-4354-8f32-1af0a20265bc","Type":"ContainerDied","Data":"92a16307c86a28fc70d9529f86967af3ca214341dd28018ba40e65faa5fda4e5"}
Jul 01 21:05:22.902644 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.902556    2573 scope.go:117] "RemoveContainer" containerID="8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4"
Jul 01 21:05:22.911758 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.911742    2573 scope.go:117] "RemoveContainer" containerID="3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62"
Jul 01 21:05:22.919085 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.919067    2573 scope.go:117] "RemoveContainer" containerID="8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4"
Jul 01 21:05:22.919320 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:05:22.919303    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4\": container with ID starting with 8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4 not found: ID does not exist" containerID="8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4"
Jul 01 21:05:22.919387 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.919329    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4"} err="failed to get container status \"8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4\": rpc error: code = NotFound desc = could not find container \"8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4\": container with ID starting with 8ca07ac67f0de0f906f4aaaea30c192af5467263a37a65fcda9f6de53888c3d4 not found: ID does not exist"
Jul 01 21:05:22.919387 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.919352    2573 scope.go:117] "RemoveContainer" containerID="3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62"
Jul 01 21:05:22.919579 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:05:22.919561    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62\": container with ID starting with 3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62 not found: ID does not exist" containerID="3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62"
Jul 01 21:05:22.919633 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.919585    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62"} err="failed to get container status \"3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62\": rpc error: code = NotFound desc = could not find container \"3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62\": container with ID starting with 3e3a67c9ef74514206af019d22d21883267674fbdd5bb4ee66b04b041e44cc62 not found: ID does not exist"
Jul 01 21:05:22.955158 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.955136    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"]
Jul 01 21:05:22.964921 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:22.964903    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts"]
Jul 01 21:05:23.590494 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:23.590462    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37768643-fcba-4354-8f32-1af0a20265bc" path="/var/lib/kubelet/pods/37768643-fcba-4354-8f32-1af0a20265bc/volumes"
Jul 01 21:05:37.189236 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.189128    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" containerID="cri-o://59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5" gracePeriod=45
Jul 01 21:05:37.321521 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.321484    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 01 21:05:37.845705 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.845683    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:05:37.956894 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.956811    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tmp-dir\") pod \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") "
Jul 01 21:05:37.956894 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.956842    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kserve-provision-location\") pod \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") "
Jul 01 21:05:37.957092 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.956898    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-home\") pod \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") "
Jul 01 21:05:37.957092 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.956935    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-model-cache\") pod \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") "
Jul 01 21:05:37.957092 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.956973    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tls-certs\") pod \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") "
Jul 01 21:05:37.957092 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.956997    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-dshm\") pod \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") "
Jul 01 21:05:37.957092 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.957032    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndp7w\" (UniqueName: \"kubernetes.io/projected/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kube-api-access-ndp7w\") pod \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\" (UID: \"83e029cf-6e76-48b1-aeb6-baac8aa2e876\") "
Jul 01 21:05:37.957342 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.957190    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-model-cache" (OuterVolumeSpecName: "model-cache") pod "83e029cf-6e76-48b1-aeb6-baac8aa2e876" (UID: "83e029cf-6e76-48b1-aeb6-baac8aa2e876"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:05:37.957342 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.957284    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:37.957589 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.957563    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-home" (OuterVolumeSpecName: "home") pod "83e029cf-6e76-48b1-aeb6-baac8aa2e876" (UID: "83e029cf-6e76-48b1-aeb6-baac8aa2e876"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:05:37.959593 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.959563    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83e029cf-6e76-48b1-aeb6-baac8aa2e876" (UID: "83e029cf-6e76-48b1-aeb6-baac8aa2e876"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:05:37.959680 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.959601    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-dshm" (OuterVolumeSpecName: "dshm") pod "83e029cf-6e76-48b1-aeb6-baac8aa2e876" (UID: "83e029cf-6e76-48b1-aeb6-baac8aa2e876"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:05:37.959680 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.959666    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kube-api-access-ndp7w" (OuterVolumeSpecName: "kube-api-access-ndp7w") pod "83e029cf-6e76-48b1-aeb6-baac8aa2e876" (UID: "83e029cf-6e76-48b1-aeb6-baac8aa2e876"). InnerVolumeSpecName "kube-api-access-ndp7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:05:37.969593 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.969504    2573 generic.go:356] "Generic (PLEG): container finished" podID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerID="59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5" exitCode=0
Jul 01 21:05:37.969696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.969632    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"
Jul 01 21:05:37.969754 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.969722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" event={"ID":"83e029cf-6e76-48b1-aeb6-baac8aa2e876","Type":"ContainerDied","Data":"59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5"}
Jul 01 21:05:37.969804 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.969787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj" event={"ID":"83e029cf-6e76-48b1-aeb6-baac8aa2e876","Type":"ContainerDied","Data":"4e60d2bedb55df5926421d2bfcb9930f2d84b8ee44c75d848cf7f80f260b5a5a"}
Jul 01 21:05:37.969871 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.969819    2573 scope.go:117] "RemoveContainer" containerID="59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5"
Jul 01 21:05:37.971119 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.971095    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83e029cf-6e76-48b1-aeb6-baac8aa2e876" (UID: "83e029cf-6e76-48b1-aeb6-baac8aa2e876"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:05:37.980702 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.980678    2573 scope.go:117] "RemoveContainer" containerID="c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833"
Jul 01 21:05:37.988260 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.988242    2573 scope.go:117] "RemoveContainer" containerID="59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5"
Jul 01 21:05:37.988512 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:05:37.988494    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5\": container with ID starting with 59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5 not found: ID does not exist" containerID="59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5"
Jul 01 21:05:37.988570 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.988519    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5"} err="failed to get container status \"59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5\": rpc error: code = NotFound desc = could not find container \"59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5\": container with ID starting with 59af3de0d0453d5e2ca445eac9ae93e41a528b74261c243ed41e2f2a08690ab5 not found: ID does not exist"
Jul 01 21:05:37.988570 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.988534    2573 scope.go:117] "RemoveContainer" containerID="c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833"
Jul 01 21:05:37.988742 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:05:37.988725    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833\": container with ID starting with c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833 not found: ID does not exist" containerID="c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833"
Jul 01 21:05:37.988799 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:37.988745    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833"} err="failed to get container status \"c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833\": rpc error: code = NotFound desc = could not find container \"c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833\": container with ID starting with c696b485c7ffd5d60f51f5431a4acc3ad0fb09e3262488d35a168930d0fe1833 not found: ID does not exist"
Jul 01 21:05:38.021716 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:38.021677    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83e029cf-6e76-48b1-aeb6-baac8aa2e876" (UID: "83e029cf-6e76-48b1-aeb6-baac8aa2e876"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:05:38.058581 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:38.058555    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:38.058733 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:38.058583    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:38.058733 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:38.058598    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:38.058733 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:38.058611    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndp7w\" (UniqueName: \"kubernetes.io/projected/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kube-api-access-ndp7w\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:38.058733 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:38.058626    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:38.058733 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:38.058640    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e029cf-6e76-48b1-aeb6-baac8aa2e876-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:05:38.300389 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:38.300359    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"]
Jul 01 21:05:38.303546 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:38.303520    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj"]
Jul 01 21:05:39.591270 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:39.591240    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" path="/var/lib/kubelet/pods/83e029cf-6e76-48b1-aeb6-baac8aa2e876/volumes"
Jul 01 21:05:51.307506 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.307465    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"]
Jul 01 21:05:51.308022 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.307969    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148e1ee0-b697-4edd-a645-cb60c5e7b4d5" containerName="main"
Jul 01 21:05:51.308022 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.307990    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="148e1ee0-b697-4edd-a645-cb60c5e7b4d5" containerName="main"
Jul 01 21:05:51.308022 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308009    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37768643-fcba-4354-8f32-1af0a20265bc" containerName="main"
Jul 01 21:05:51.308022 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308017    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37768643-fcba-4354-8f32-1af0a20265bc" containerName="main"
Jul 01 21:05:51.308237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308034    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main"
Jul 01 21:05:51.308237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308042    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main"
Jul 01 21:05:51.308237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308053    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37768643-fcba-4354-8f32-1af0a20265bc" containerName="storage-initializer"
Jul 01 21:05:51.308237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308061    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37768643-fcba-4354-8f32-1af0a20265bc" containerName="storage-initializer"
Jul 01 21:05:51.308237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308073    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="storage-initializer"
Jul 01 21:05:51.308237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308082    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="storage-initializer"
Jul 01 21:05:51.308237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308183    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="148e1ee0-b697-4edd-a645-cb60c5e7b4d5" containerName="main"
Jul 01 21:05:51.308237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308195    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37768643-fcba-4354-8f32-1af0a20265bc" containerName="main"
Jul 01 21:05:51.308237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.308206    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e029cf-6e76-48b1-aeb6-baac8aa2e876" containerName="main"
Jul 01 21:05:51.312902 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.312880    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.316025 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.316006    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-76tz7\""
Jul 01 21:05:51.316119 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.316026    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 21:05:51.316293 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.316277    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 01 21:05:51.316382 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.316364    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 21:05:51.317165 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.317151    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:05:51.329896 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.329877    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"]
Jul 01 21:05:51.349587 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.349567    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-tmp-dir\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.349676 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.349609    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jdnr\" (UniqueName: \"kubernetes.io/projected/851d695c-a27a-49e3-b0ae-4370cf7086f3-kube-api-access-6jdnr\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.349676 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.349638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-kserve-provision-location\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.349798 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.349697    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-home\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.349798 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.349717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d695c-a27a-49e3-b0ae-4370cf7086f3-tls-certs\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.349798 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.349734    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-model-cache\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.349798 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.349759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-dshm\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450151 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-home\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450264 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450158    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d695c-a27a-49e3-b0ae-4370cf7086f3-tls-certs\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450264 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450175    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-model-cache\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450264 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450201    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-dshm\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450264 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-tmp-dir\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450468 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450277    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jdnr\" (UniqueName: \"kubernetes.io/projected/851d695c-a27a-49e3-b0ae-4370cf7086f3-kube-api-access-6jdnr\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450468 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450313    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-kserve-provision-location\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450637 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450564    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-home\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450637 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450584    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-model-cache\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450816 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450660    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-kserve-provision-location\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.450816 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.450678    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-tmp-dir\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.452577 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.452559    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-dshm\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.452879 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.452843    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d695c-a27a-49e3-b0ae-4370cf7086f3-tls-certs\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.460948 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.460918    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jdnr\" (UniqueName: \"kubernetes.io/projected/851d695c-a27a-49e3-b0ae-4370cf7086f3-kube-api-access-6jdnr\") pod \"auth-disabled-test-kserve-68bf46444d-nz2fk\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.622558 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.622528    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:05:51.779568 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:05:51.779542    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851d695c_a27a_49e3_b0ae_4370cf7086f3.slice/crio-8e33a1c98d450311ae6decd8edd50c5901bd2c96ee985b70bf41c0722ee165ed WatchSource:0}: Error finding container 8e33a1c98d450311ae6decd8edd50c5901bd2c96ee985b70bf41c0722ee165ed: Status 404 returned error can't find the container with id 8e33a1c98d450311ae6decd8edd50c5901bd2c96ee985b70bf41c0722ee165ed
Jul 01 21:05:51.785208 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.785186    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"]
Jul 01 21:05:51.862187 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.862157    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"]
Jul 01 21:05:51.865380 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.865363    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:51.868917 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.868896    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-hg48w\""
Jul 01 21:05:51.888381 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.888326    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"]
Jul 01 21:05:51.954231 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.954200    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:51.954398 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.954243    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffsj\" (UniqueName: \"kubernetes.io/projected/b0572344-28fe-43a3-8247-04651ae33549-kube-api-access-cffsj\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:51.954398 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.954277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0572344-28fe-43a3-8247-04651ae33549-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:51.954398 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:51.954322    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.033197 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.033160    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" event={"ID":"851d695c-a27a-49e3-b0ae-4370cf7086f3","Type":"ContainerStarted","Data":"e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682"}
Jul 01 21:05:52.033321 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.033212    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" event={"ID":"851d695c-a27a-49e3-b0ae-4370cf7086f3","Type":"ContainerStarted","Data":"8e33a1c98d450311ae6decd8edd50c5901bd2c96ee985b70bf41c0722ee165ed"}
Jul 01 21:05:52.055533 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.055505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.055635 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.055540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffsj\" (UniqueName: \"kubernetes.io/projected/b0572344-28fe-43a3-8247-04651ae33549-kube-api-access-cffsj\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.055635 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.055568    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0572344-28fe-43a3-8247-04651ae33549-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.055635 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.055614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.055973 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.055947    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.056061 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.055959    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.058037 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.058017    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0572344-28fe-43a3-8247-04651ae33549-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.067308 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.067284    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffsj\" (UniqueName: \"kubernetes.io/projected/b0572344-28fe-43a3-8247-04651ae33549-kube-api-access-cffsj\") pod \"auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.173947 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.173876    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:52.307955 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:52.307932    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"]
Jul 01 21:05:52.309445 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:05:52.309409    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0572344_28fe_43a3_8247_04651ae33549.slice/crio-aff9588b49d721ac1a432662aab96f9b03050cc59ec1b39d83704566de48865b WatchSource:0}: Error finding container aff9588b49d721ac1a432662aab96f9b03050cc59ec1b39d83704566de48865b: Status 404 returned error can't find the container with id aff9588b49d721ac1a432662aab96f9b03050cc59ec1b39d83704566de48865b
Jul 01 21:05:53.042315 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:53.042276    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" event={"ID":"b0572344-28fe-43a3-8247-04651ae33549","Type":"ContainerStarted","Data":"c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343"}
Jul 01 21:05:53.042540 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:53.042339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" event={"ID":"b0572344-28fe-43a3-8247-04651ae33549","Type":"ContainerStarted","Data":"aff9588b49d721ac1a432662aab96f9b03050cc59ec1b39d83704566de48865b"}
Jul 01 21:05:54.047751 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:54.047716    2573 generic.go:356] "Generic (PLEG): container finished" podID="b0572344-28fe-43a3-8247-04651ae33549" containerID="c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343" exitCode=0
Jul 01 21:05:54.048196 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:54.047804    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" event={"ID":"b0572344-28fe-43a3-8247-04651ae33549","Type":"ContainerDied","Data":"c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343"}
Jul 01 21:05:55.055822 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:55.055785    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" event={"ID":"b0572344-28fe-43a3-8247-04651ae33549","Type":"ContainerStarted","Data":"9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787"}
Jul 01 21:05:55.056627 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:55.055893    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:05:55.084473 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:55.084426    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" podStartSLOduration=3.399182583 podStartE2EDuration="4.084412288s" totalImagesPullingTime="0s" totalInitContainerRuntime="685.229705ms" isStatefulPod=false podCreationTimestamp="2026-07-01 21:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:05:55.082036488 +0000 UTC m=+1187.993371116" watchObservedRunningTime="2026-07-01 21:05:55.084412288 +0000 UTC m=+1187.995746941"
Jul 01 21:05:56.061373 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:56.061275    2573 generic.go:356] "Generic (PLEG): container finished" podID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerID="e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682" exitCode=0
Jul 01 21:05:56.061373 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:56.061343    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" event={"ID":"851d695c-a27a-49e3-b0ae-4370cf7086f3","Type":"ContainerDied","Data":"e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682"}
Jul 01 21:05:57.067388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:57.067353    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" event={"ID":"851d695c-a27a-49e3-b0ae-4370cf7086f3","Type":"ContainerStarted","Data":"0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946"}
Jul 01 21:05:57.094865 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:57.094817    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podStartSLOduration=2.277330763 podStartE2EDuration="6.094801401s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.817470638s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:05:57.09414094 +0000 UTC m=+1190.005475568" watchObservedRunningTime="2026-07-01 21:05:57.094801401 +0000 UTC m=+1190.006136026"
Jul 01 21:05:57.623526 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:05:57.623494    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:06:01.623130 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:01.623093    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:06:01.624652 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:01.624618    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:06:07.592207 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:07.592177    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:06:07.593686 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:07.593667    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:06:07.599577 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:07.599561    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:06:07.601004 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:07.600985    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:06:11.623171 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:11.623130    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:06:21.623315 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:21.623271    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:06:26.062831 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:26.062805    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:06:31.623551 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:31.623504    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:06:41.623789 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:41.623746    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:06:51.623121 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:06:51.623084    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:07:01.624006 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:01.623913    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:07:11.623752 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:11.623712    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:07:21.623622 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:21.623581    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:07:31.623591 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:31.623552    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:07:41.623864 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:41.623805    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 01 21:07:51.633656 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:51.633626    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:07:51.641599 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:51.641573    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:07:53.093174 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.093137    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"]
Jul 01 21:07:53.097786 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.097768    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.100596 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.100577    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:07:53.100747 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.100640    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 01 21:07:53.111470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.111449    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"]
Jul 01 21:07:53.132038 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.132011    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62586098-ac9f-4cb3-bb0f-0095763221c4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.132166 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.132076    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.132166 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.132115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lxl\" (UniqueName: \"kubernetes.io/projected/62586098-ac9f-4cb3-bb0f-0095763221c4-kube-api-access-89lxl\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.132308 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.132178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.132308 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.132206    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.132308 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.132240    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62586098-ac9f-4cb3-bb0f-0095763221c4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.132464 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.132355    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.132464 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.132402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233146 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233259 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233154    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233259 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233177    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62586098-ac9f-4cb3-bb0f-0095763221c4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233259 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233259 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233245    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lxl\" (UniqueName: \"kubernetes.io/projected/62586098-ac9f-4cb3-bb0f-0095763221c4-kube-api-access-89lxl\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233368    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62586098-ac9f-4cb3-bb0f-0095763221c4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233637 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233513    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233637 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233562    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233756 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233661    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.233756 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.233701    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.234093 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.234073    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62586098-ac9f-4cb3-bb0f-0095763221c4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.235532 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.235510    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.235675 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.235634    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62586098-ac9f-4cb3-bb0f-0095763221c4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.241323 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.241300    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lxl\" (UniqueName: \"kubernetes.io/projected/62586098-ac9f-4cb3-bb0f-0095763221c4-kube-api-access-89lxl\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.407322 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.407244    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:07:53.542258 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.542234    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"]
Jul 01 21:07:53.543991 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:07:53.543967    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62586098_ac9f_4cb3_bb0f_0095763221c4.slice/crio-934722aae95a658387e980e0f61a6fd8ee6d2e9a5ec307b7ed0c8183469af8b3 WatchSource:0}: Error finding container 934722aae95a658387e980e0f61a6fd8ee6d2e9a5ec307b7ed0c8183469af8b3: Status 404 returned error can't find the container with id 934722aae95a658387e980e0f61a6fd8ee6d2e9a5ec307b7ed0c8183469af8b3
Jul 01 21:07:53.614297 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.614267    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" event={"ID":"62586098-ac9f-4cb3-bb0f-0095763221c4","Type":"ContainerStarted","Data":"f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246"}
Jul 01 21:07:53.614402 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:53.614318    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" event={"ID":"62586098-ac9f-4cb3-bb0f-0095763221c4","Type":"ContainerStarted","Data":"934722aae95a658387e980e0f61a6fd8ee6d2e9a5ec307b7ed0c8183469af8b3"}
Jul 01 21:07:58.639349 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:58.639260    2573 generic.go:356] "Generic (PLEG): container finished" podID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerID="f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246" exitCode=0
Jul 01 21:07:58.639349 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:58.639334    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" event={"ID":"62586098-ac9f-4cb3-bb0f-0095763221c4","Type":"ContainerDied","Data":"f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246"}
Jul 01 21:07:59.646729 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:59.646685    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" event={"ID":"62586098-ac9f-4cb3-bb0f-0095763221c4","Type":"ContainerStarted","Data":"654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562"}
Jul 01 21:07:59.688477 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:07:59.688411    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podStartSLOduration=1.9326136329999999 podStartE2EDuration="6.688388694s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.755775061s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:07:59.686083805 +0000 UTC m=+1312.597418431" watchObservedRunningTime="2026-07-01 21:07:59.688388694 +0000 UTC m=+1312.599723321"
Jul 01 21:08:00.407367 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:00.407320    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:08:01.909797 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:01.909768    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"]
Jul 01 21:08:01.915308 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:01.915269    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"]
Jul 01 21:08:01.924012 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:08:01.923973    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 21:08:01.924012 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:08:01.924012 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:08:01.924012 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787" cmd=["/bin/sleep","15"]
Jul 01 21:08:01.924297 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:01.924033    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" message="" err=<
Jul 01 21:08:01.924297 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:08:01.924297 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:08:01.924297 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 21:08:01.924297 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:08:01.924050    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 21:08:01.924297 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:08:01.924297 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:08:01.924297 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" podUID="b0572344-28fe-43a3-8247-04651ae33549" containerName="main" containerID="cri-o://9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787"
Jul 01 21:08:01.924297 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:01.924090    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" podUID="b0572344-28fe-43a3-8247-04651ae33549" containerName="main" containerID="cri-o://9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787" gracePeriod=60
Jul 01 21:08:02.180640 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.180617    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:08:02.215594 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.215558    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0572344-28fe-43a3-8247-04651ae33549-tls-certs\") pod \"b0572344-28fe-43a3-8247-04651ae33549\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") "
Jul 01 21:08:02.215734 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.215651    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cffsj\" (UniqueName: \"kubernetes.io/projected/b0572344-28fe-43a3-8247-04651ae33549-kube-api-access-cffsj\") pod \"b0572344-28fe-43a3-8247-04651ae33549\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") "
Jul 01 21:08:02.215734 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.215685    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-kserve-provision-location\") pod \"b0572344-28fe-43a3-8247-04651ae33549\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") "
Jul 01 21:08:02.215880 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.215747    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-tokenizer-uds\") pod \"b0572344-28fe-43a3-8247-04651ae33549\" (UID: \"b0572344-28fe-43a3-8247-04651ae33549\") "
Jul 01 21:08:02.216095 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.216059    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b0572344-28fe-43a3-8247-04651ae33549" (UID: "b0572344-28fe-43a3-8247-04651ae33549"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:08:02.216379 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.216356    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0572344-28fe-43a3-8247-04651ae33549" (UID: "b0572344-28fe-43a3-8247-04651ae33549"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:08:02.217647 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.217627    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0572344-28fe-43a3-8247-04651ae33549-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0572344-28fe-43a3-8247-04651ae33549" (UID: "b0572344-28fe-43a3-8247-04651ae33549"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:08:02.218132 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.218114    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0572344-28fe-43a3-8247-04651ae33549-kube-api-access-cffsj" (OuterVolumeSpecName: "kube-api-access-cffsj") pod "b0572344-28fe-43a3-8247-04651ae33549" (UID: "b0572344-28fe-43a3-8247-04651ae33549"). InnerVolumeSpecName "kube-api-access-cffsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:08:02.316807 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.316775    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0572344-28fe-43a3-8247-04651ae33549-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:02.316807 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.316802    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffsj\" (UniqueName: \"kubernetes.io/projected/b0572344-28fe-43a3-8247-04651ae33549-kube-api-access-cffsj\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:02.316807 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.316813    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:02.317051 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.316822    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0572344-28fe-43a3-8247-04651ae33549-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:02.675405 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.675377    2573 generic.go:356] "Generic (PLEG): container finished" podID="b0572344-28fe-43a3-8247-04651ae33549" containerID="9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787" exitCode=0
Jul 01 21:08:02.675569 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.675439    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"
Jul 01 21:08:02.675569 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.675465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" event={"ID":"b0572344-28fe-43a3-8247-04651ae33549","Type":"ContainerDied","Data":"9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787"}
Jul 01 21:08:02.675569 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.675514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5" event={"ID":"b0572344-28fe-43a3-8247-04651ae33549","Type":"ContainerDied","Data":"aff9588b49d721ac1a432662aab96f9b03050cc59ec1b39d83704566de48865b"}
Jul 01 21:08:02.675569 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.675535    2573 scope.go:117] "RemoveContainer" containerID="9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787"
Jul 01 21:08:02.684143 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.684125    2573 scope.go:117] "RemoveContainer" containerID="c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343"
Jul 01 21:08:02.692007 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.691990    2573 scope.go:117] "RemoveContainer" containerID="9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787"
Jul 01 21:08:02.692251 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:08:02.692232    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787\": container with ID starting with 9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787 not found: ID does not exist" containerID="9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787"
Jul 01 21:08:02.692300 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.692259    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787"} err="failed to get container status \"9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787\": rpc error: code = NotFound desc = could not find container \"9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787\": container with ID starting with 9b7b6110a9f553d27ebf46a37238e99fb621787ccdff8c87e9a2ceab43e1d787 not found: ID does not exist"
Jul 01 21:08:02.692300 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.692277    2573 scope.go:117] "RemoveContainer" containerID="c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343"
Jul 01 21:08:02.692485 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:08:02.692471    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343\": container with ID starting with c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343 not found: ID does not exist" containerID="c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343"
Jul 01 21:08:02.692529 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.692487    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343"} err="failed to get container status \"c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343\": rpc error: code = NotFound desc = could not find container \"c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343\": container with ID starting with c371ae0431b43ee3027504a0113efc85a35b2d93e3a16ecd44f371864d9e7343 not found: ID does not exist"
Jul 01 21:08:02.702031 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.702010    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"]
Jul 01 21:08:02.705040 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:02.705021    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5"]
Jul 01 21:08:03.408336 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:03.408304    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:08:03.410009 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:03.409966    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 01 21:08:03.590742 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:03.590703    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0572344-28fe-43a3-8247-04651ae33549" path="/var/lib/kubelet/pods/b0572344-28fe-43a3-8247-04651ae33549/volumes"
Jul 01 21:08:13.408682 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:13.408631    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 01 21:08:16.935167 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:16.935127    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" containerID="cri-o://0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946" gracePeriod=45
Jul 01 21:08:17.684673 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.684649    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:08:17.756845 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.756761    2573 generic.go:356] "Generic (PLEG): container finished" podID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerID="0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946" exitCode=0
Jul 01 21:08:17.756845 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.756807    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"
Jul 01 21:08:17.756845 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.756841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" event={"ID":"851d695c-a27a-49e3-b0ae-4370cf7086f3","Type":"ContainerDied","Data":"0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946"}
Jul 01 21:08:17.757157 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.756914    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" event={"ID":"851d695c-a27a-49e3-b0ae-4370cf7086f3","Type":"ContainerDied","Data":"8e33a1c98d450311ae6decd8edd50c5901bd2c96ee985b70bf41c0722ee165ed"}
Jul 01 21:08:17.757157 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.756957    2573 scope.go:117] "RemoveContainer" containerID="0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946"
Jul 01 21:08:17.766984 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.766965    2573 scope.go:117] "RemoveContainer" containerID="e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682"
Jul 01 21:08:17.774947 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.774927    2573 scope.go:117] "RemoveContainer" containerID="0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946"
Jul 01 21:08:17.775183 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:08:17.775164    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946\": container with ID starting with 0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946 not found: ID does not exist" containerID="0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946"
Jul 01 21:08:17.775247 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.775191    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946"} err="failed to get container status \"0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946\": rpc error: code = NotFound desc = could not find container \"0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946\": container with ID starting with 0d6c3dae9c4aaccc40cfc8fdf4dc8c6d9df72c21261ab935dd3087b755411946 not found: ID does not exist"
Jul 01 21:08:17.775247 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.775208    2573 scope.go:117] "RemoveContainer" containerID="e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682"
Jul 01 21:08:17.775441 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:08:17.775422    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682\": container with ID starting with e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682 not found: ID does not exist" containerID="e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682"
Jul 01 21:08:17.775506 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.775446    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682"} err="failed to get container status \"e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682\": rpc error: code = NotFound desc = could not find container \"e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682\": container with ID starting with e68386e891885c4204e0004d1a8c81f337be144456ba850e55cb710bff5a4682 not found: ID does not exist"
Jul 01 21:08:17.844842 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.844810    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d695c-a27a-49e3-b0ae-4370cf7086f3-tls-certs\") pod \"851d695c-a27a-49e3-b0ae-4370cf7086f3\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") "
Jul 01 21:08:17.845015 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.844886    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-kserve-provision-location\") pod \"851d695c-a27a-49e3-b0ae-4370cf7086f3\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") "
Jul 01 21:08:17.845015 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.844937    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-model-cache\") pod \"851d695c-a27a-49e3-b0ae-4370cf7086f3\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") "
Jul 01 21:08:17.845015 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.844973    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-tmp-dir\") pod \"851d695c-a27a-49e3-b0ae-4370cf7086f3\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") "
Jul 01 21:08:17.845015 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.844999    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-home\") pod \"851d695c-a27a-49e3-b0ae-4370cf7086f3\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") "
Jul 01 21:08:17.845232 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.845039    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-dshm\") pod \"851d695c-a27a-49e3-b0ae-4370cf7086f3\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") "
Jul 01 21:08:17.845232 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.845078    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jdnr\" (UniqueName: \"kubernetes.io/projected/851d695c-a27a-49e3-b0ae-4370cf7086f3-kube-api-access-6jdnr\") pod \"851d695c-a27a-49e3-b0ae-4370cf7086f3\" (UID: \"851d695c-a27a-49e3-b0ae-4370cf7086f3\") "
Jul 01 21:08:17.847343 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.845795    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "851d695c-a27a-49e3-b0ae-4370cf7086f3" (UID: "851d695c-a27a-49e3-b0ae-4370cf7086f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:08:17.847343 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.846611    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-home" (OuterVolumeSpecName: "home") pod "851d695c-a27a-49e3-b0ae-4370cf7086f3" (UID: "851d695c-a27a-49e3-b0ae-4370cf7086f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:08:17.850873 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.849227    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851d695c-a27a-49e3-b0ae-4370cf7086f3-kube-api-access-6jdnr" (OuterVolumeSpecName: "kube-api-access-6jdnr") pod "851d695c-a27a-49e3-b0ae-4370cf7086f3" (UID: "851d695c-a27a-49e3-b0ae-4370cf7086f3"). InnerVolumeSpecName "kube-api-access-6jdnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:08:17.852897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.852218    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851d695c-a27a-49e3-b0ae-4370cf7086f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "851d695c-a27a-49e3-b0ae-4370cf7086f3" (UID: "851d695c-a27a-49e3-b0ae-4370cf7086f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:08:17.853272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.853246    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-dshm" (OuterVolumeSpecName: "dshm") pod "851d695c-a27a-49e3-b0ae-4370cf7086f3" (UID: "851d695c-a27a-49e3-b0ae-4370cf7086f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:08:17.860074 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.860035    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "851d695c-a27a-49e3-b0ae-4370cf7086f3" (UID: "851d695c-a27a-49e3-b0ae-4370cf7086f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:08:17.918179 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.918148    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "851d695c-a27a-49e3-b0ae-4370cf7086f3" (UID: "851d695c-a27a-49e3-b0ae-4370cf7086f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:08:17.946007 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.945979    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:17.946007 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.946004    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:17.946330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.946017    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:17.946330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.946028    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jdnr\" (UniqueName: \"kubernetes.io/projected/851d695c-a27a-49e3-b0ae-4370cf7086f3-kube-api-access-6jdnr\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:17.946330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.946039    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851d695c-a27a-49e3-b0ae-4370cf7086f3-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:17.946330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.946052    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:17.946330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:17.946065    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851d695c-a27a-49e3-b0ae-4370cf7086f3-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:08:18.081958 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:18.081933    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"]
Jul 01 21:08:18.088948 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:18.088916    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk"]
Jul 01 21:08:18.634980 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:18.634938    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 01 21:08:19.591683 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:19.591650    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" path="/var/lib/kubelet/pods/851d695c-a27a-49e3-b0ae-4370cf7086f3/volumes"
Jul 01 21:08:23.408523 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:23.408481    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 01 21:08:33.408045 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:33.407951    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 01 21:08:43.407843 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:43.407798    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 01 21:08:53.408067 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:08:53.408013    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 01 21:09:03.408504 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:03.408462    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 01 21:09:07.493091 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493039    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"]
Jul 01 21:09:07.493653 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493620    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="storage-initializer"
Jul 01 21:09:07.493740 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493659    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="storage-initializer"
Jul 01 21:09:07.493740 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493676    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0572344-28fe-43a3-8247-04651ae33549" containerName="storage-initializer"
Jul 01 21:09:07.493740 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493685    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0572344-28fe-43a3-8247-04651ae33549" containerName="storage-initializer"
Jul 01 21:09:07.493740 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493710    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0572344-28fe-43a3-8247-04651ae33549" containerName="main"
Jul 01 21:09:07.493740 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493719    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0572344-28fe-43a3-8247-04651ae33549" containerName="main"
Jul 01 21:09:07.494085 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493748    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main"
Jul 01 21:09:07.494085 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493757    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main"
Jul 01 21:09:07.494085 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493882    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0572344-28fe-43a3-8247-04651ae33549" containerName="main"
Jul 01 21:09:07.494085 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.493893    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="851d695c-a27a-49e3-b0ae-4370cf7086f3" containerName="main"
Jul 01 21:09:07.496660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.496636    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.499711 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.499672    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 01 21:09:07.507204 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.507036    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"]
Jul 01 21:09:07.558396 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.558363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.558396 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.558394    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.558659 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.558418    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.558659 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.558443    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.558659 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.558531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b65003-4119-43ed-9670-23685bacbba9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.558659 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.558597    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.558659 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.558632    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9hnn\" (UniqueName: \"kubernetes.io/projected/87b65003-4119-43ed-9670-23685bacbba9-kube-api-access-m9hnn\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.659364 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.659300    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.659621 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.659505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.659702 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.659616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.659702 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.659684    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.659847 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.659765    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b65003-4119-43ed-9670-23685bacbba9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.659961 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.659901    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.660223 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.660194    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.660306 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.660232    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.660388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.660349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.660450 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.660388    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9hnn\" (UniqueName: \"kubernetes.io/projected/87b65003-4119-43ed-9670-23685bacbba9-kube-api-access-m9hnn\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.660978 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.660922    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.663228 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.663197    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.663580 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.663557    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b65003-4119-43ed-9670-23685bacbba9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.670130 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.670108    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9hnn\" (UniqueName: \"kubernetes.io/projected/87b65003-4119-43ed-9670-23685bacbba9-kube-api-access-m9hnn\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.812730 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.812656    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:07.980492 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:07.980470    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"]
Jul 01 21:09:07.982773 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:09:07.982747    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b65003_4119_43ed_9670_23685bacbba9.slice/crio-59ce823cbb154be50c762164c6c7f21d33ab83b1c3c1e5979cd8617098aef8e8 WatchSource:0}: Error finding container 59ce823cbb154be50c762164c6c7f21d33ab83b1c3c1e5979cd8617098aef8e8: Status 404 returned error can't find the container with id 59ce823cbb154be50c762164c6c7f21d33ab83b1c3c1e5979cd8617098aef8e8
Jul 01 21:09:08.989847 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:08.989788    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" event={"ID":"87b65003-4119-43ed-9670-23685bacbba9","Type":"ContainerStarted","Data":"b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183"}
Jul 01 21:09:08.990948 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:08.989885    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" event={"ID":"87b65003-4119-43ed-9670-23685bacbba9","Type":"ContainerStarted","Data":"59ce823cbb154be50c762164c6c7f21d33ab83b1c3c1e5979cd8617098aef8e8"}
Jul 01 21:09:13.006949 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:13.006915    2573 generic.go:356] "Generic (PLEG): container finished" podID="87b65003-4119-43ed-9670-23685bacbba9" containerID="b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183" exitCode=0
Jul 01 21:09:13.006949 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:13.006952    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" event={"ID":"87b65003-4119-43ed-9670-23685bacbba9","Type":"ContainerDied","Data":"b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183"}
Jul 01 21:09:13.424125 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:13.424093    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:09:13.436608 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:13.436581    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:09:14.014026 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:14.013995    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" event={"ID":"87b65003-4119-43ed-9670-23685bacbba9","Type":"ContainerStarted","Data":"34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2"}
Jul 01 21:09:14.038418 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:14.038365    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podStartSLOduration=3.084050457 podStartE2EDuration="7.038343768s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.954293311s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:09:14.036515229 +0000 UTC m=+1386.947849856" watchObservedRunningTime="2026-07-01 21:09:14.038343768 +0000 UTC m=+1386.949678396"
Jul 01 21:09:14.813361 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:14.813331    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:17.813515 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:17.813477    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:09:17.815087 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:17.815049    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:09:27.813765 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:27.813709    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:09:37.813145 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:37.813103    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:09:44.928210 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:44.928173    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"]
Jul 01 21:09:47.813660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:47.813609    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:09:57.814072 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:57.814027    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:09:59.954197 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:09:59.954079    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main" containerID="cri-o://654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562" gracePeriod=45
Jul 01 21:10:00.425126 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.425085    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Jul 01 21:10:00.601515 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.601492    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:10:00.714066 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.713996    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62586098-ac9f-4cb3-bb0f-0095763221c4-cabundle-cert\") pod \"62586098-ac9f-4cb3-bb0f-0095763221c4\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") "
Jul 01 21:10:00.714066 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714034    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-tmp-dir\") pod \"62586098-ac9f-4cb3-bb0f-0095763221c4\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") "
Jul 01 21:10:00.714066 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714064    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62586098-ac9f-4cb3-bb0f-0095763221c4-tls-certs\") pod \"62586098-ac9f-4cb3-bb0f-0095763221c4\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") "
Jul 01 21:10:00.714323 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714097    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-kserve-provision-location\") pod \"62586098-ac9f-4cb3-bb0f-0095763221c4\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") "
Jul 01 21:10:00.714323 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714126    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-home\") pod \"62586098-ac9f-4cb3-bb0f-0095763221c4\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") "
Jul 01 21:10:00.714323 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714151    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89lxl\" (UniqueName: \"kubernetes.io/projected/62586098-ac9f-4cb3-bb0f-0095763221c4-kube-api-access-89lxl\") pod \"62586098-ac9f-4cb3-bb0f-0095763221c4\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") "
Jul 01 21:10:00.714323 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714176    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-model-cache\") pod \"62586098-ac9f-4cb3-bb0f-0095763221c4\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") "
Jul 01 21:10:00.714323 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714232    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-dshm\") pod \"62586098-ac9f-4cb3-bb0f-0095763221c4\" (UID: \"62586098-ac9f-4cb3-bb0f-0095763221c4\") "
Jul 01 21:10:00.714580 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714420    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62586098-ac9f-4cb3-bb0f-0095763221c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62586098-ac9f-4cb3-bb0f-0095763221c4" (UID: "62586098-ac9f-4cb3-bb0f-0095763221c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:10:00.714642 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714574    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-home" (OuterVolumeSpecName: "home") pod "62586098-ac9f-4cb3-bb0f-0095763221c4" (UID: "62586098-ac9f-4cb3-bb0f-0095763221c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:10:00.714792 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.714765    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "62586098-ac9f-4cb3-bb0f-0095763221c4" (UID: "62586098-ac9f-4cb3-bb0f-0095763221c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:10:00.716683 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.716653    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62586098-ac9f-4cb3-bb0f-0095763221c4-kube-api-access-89lxl" (OuterVolumeSpecName: "kube-api-access-89lxl") pod "62586098-ac9f-4cb3-bb0f-0095763221c4" (UID: "62586098-ac9f-4cb3-bb0f-0095763221c4"). InnerVolumeSpecName "kube-api-access-89lxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:10:00.716818 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.716772    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-dshm" (OuterVolumeSpecName: "dshm") pod "62586098-ac9f-4cb3-bb0f-0095763221c4" (UID: "62586098-ac9f-4cb3-bb0f-0095763221c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:10:00.717046 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.717015    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62586098-ac9f-4cb3-bb0f-0095763221c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62586098-ac9f-4cb3-bb0f-0095763221c4" (UID: "62586098-ac9f-4cb3-bb0f-0095763221c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:10:00.725312 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.725286    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62586098-ac9f-4cb3-bb0f-0095763221c4" (UID: "62586098-ac9f-4cb3-bb0f-0095763221c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:10:00.771682 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.771652    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62586098-ac9f-4cb3-bb0f-0095763221c4" (UID: "62586098-ac9f-4cb3-bb0f-0095763221c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:10:00.815689 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.815664    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62586098-ac9f-4cb3-bb0f-0095763221c4-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:10:00.815689 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.815693    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:10:00.815844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.815703    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:10:00.815844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.815712    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lxl\" (UniqueName: \"kubernetes.io/projected/62586098-ac9f-4cb3-bb0f-0095763221c4-kube-api-access-89lxl\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:10:00.815844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.815722    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:10:00.815844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.815730    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:10:00.815844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.815737    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62586098-ac9f-4cb3-bb0f-0095763221c4-cabundle-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:10:00.815844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:00.815770    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62586098-ac9f-4cb3-bb0f-0095763221c4-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:10:01.257447 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.257416    2573 generic.go:356] "Generic (PLEG): container finished" podID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerID="654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562" exitCode=0
Jul 01 21:10:01.257834 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.257473    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" event={"ID":"62586098-ac9f-4cb3-bb0f-0095763221c4","Type":"ContainerDied","Data":"654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562"}
Jul 01 21:10:01.257834 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.257489    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"
Jul 01 21:10:01.257834 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.257538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl" event={"ID":"62586098-ac9f-4cb3-bb0f-0095763221c4","Type":"ContainerDied","Data":"934722aae95a658387e980e0f61a6fd8ee6d2e9a5ec307b7ed0c8183469af8b3"}
Jul 01 21:10:01.257834 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.257562    2573 scope.go:117] "RemoveContainer" containerID="654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562"
Jul 01 21:10:01.266033 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.266016    2573 scope.go:117] "RemoveContainer" containerID="f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246"
Jul 01 21:10:01.274209 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.274193    2573 scope.go:117] "RemoveContainer" containerID="654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562"
Jul 01 21:10:01.274486 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:10:01.274465    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562\": container with ID starting with 654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562 not found: ID does not exist" containerID="654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562"
Jul 01 21:10:01.274564 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.274492    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562"} err="failed to get container status \"654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562\": rpc error: code = NotFound desc = could not find container \"654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562\": container with ID starting with 654f3139f8c6ec9870a7c1f023cc4c08c1cb6b1a68c53c3224aa3f68035d4562 not found: ID does not exist"
Jul 01 21:10:01.274564 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.274509    2573 scope.go:117] "RemoveContainer" containerID="f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246"
Jul 01 21:10:01.274720 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:10:01.274704    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246\": container with ID starting with f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246 not found: ID does not exist" containerID="f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246"
Jul 01 21:10:01.274791 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.274721    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246"} err="failed to get container status \"f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246\": rpc error: code = NotFound desc = could not find container \"f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246\": container with ID starting with f5be95dce7d21d68e0b1d7f02396e5c46ff9705bbf233824abbaa75a4c18a246 not found: ID does not exist"
Jul 01 21:10:01.285716 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.285692    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"]
Jul 01 21:10:01.290896 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.290872    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl"]
Jul 01 21:10:01.591986 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:01.591950    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" path="/var/lib/kubelet/pods/62586098-ac9f-4cb3-bb0f-0095763221c4/volumes"
Jul 01 21:10:07.813710 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:07.813674    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:10:12.166161 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.166124    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"]
Jul 01 21:10:12.166632 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.166614    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="storage-initializer"
Jul 01 21:10:12.166687 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.166637    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="storage-initializer"
Jul 01 21:10:12.166687 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.166652    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main"
Jul 01 21:10:12.166687 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.166660    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main"
Jul 01 21:10:12.166849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.166748    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62586098-ac9f-4cb3-bb0f-0095763221c4" containerName="main"
Jul 01 21:10:12.169724 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.169702    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.172707 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.172687    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 01 21:10:12.172826 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.172719    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:10:12.180969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.180948    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"]
Jul 01 21:10:12.311285 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.311255    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.311465 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.311299    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1c2ddd-1999-4917-9046-dc608949bf09-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.311465 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.311391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.311465 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.311428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.311642 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.311466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c2ddd-1999-4917-9046-dc608949bf09-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.311642 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.311532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2sz\" (UniqueName: \"kubernetes.io/projected/8b1c2ddd-1999-4917-9046-dc608949bf09-kube-api-access-zx2sz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.311642 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.311582    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.311642 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.311610    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.412440 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.412404    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1c2ddd-1999-4917-9046-dc608949bf09-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.412610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.412473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.412610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.412502    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.412610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.412539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c2ddd-1999-4917-9046-dc608949bf09-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.412610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.412579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2sz\" (UniqueName: \"kubernetes.io/projected/8b1c2ddd-1999-4917-9046-dc608949bf09-kube-api-access-zx2sz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.412849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.412614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.412849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.412640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.412849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.412687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.413134 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.413110    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.413478 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.413443    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.413575 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.413487    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.413646 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.413608    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c2ddd-1999-4917-9046-dc608949bf09-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.413833 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.413810    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.415389 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.415367    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.415474 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.415440    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1c2ddd-1999-4917-9046-dc608949bf09-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.427758 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.427703    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2sz\" (UniqueName: \"kubernetes.io/projected/8b1c2ddd-1999-4917-9046-dc608949bf09-kube-api-access-zx2sz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.480742 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.480712    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:12.621270 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:12.621120    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"]
Jul 01 21:10:12.623875 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:10:12.623837    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1c2ddd_1999_4917_9046_dc608949bf09.slice/crio-aece49db9eb73661a27d056874c29cc01a3a85408864cc7861835a033b33189b WatchSource:0}: Error finding container aece49db9eb73661a27d056874c29cc01a3a85408864cc7861835a033b33189b: Status 404 returned error can't find the container with id aece49db9eb73661a27d056874c29cc01a3a85408864cc7861835a033b33189b
Jul 01 21:10:13.321448 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:13.321408    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" event={"ID":"8b1c2ddd-1999-4917-9046-dc608949bf09","Type":"ContainerStarted","Data":"5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01"}
Jul 01 21:10:13.321830 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:13.321467    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" event={"ID":"8b1c2ddd-1999-4917-9046-dc608949bf09","Type":"ContainerStarted","Data":"aece49db9eb73661a27d056874c29cc01a3a85408864cc7861835a033b33189b"}
Jul 01 21:10:17.341224 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:17.341191    2573 generic.go:356] "Generic (PLEG): container finished" podID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerID="5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01" exitCode=0
Jul 01 21:10:17.341599 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:17.341269    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" event={"ID":"8b1c2ddd-1999-4917-9046-dc608949bf09","Type":"ContainerDied","Data":"5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01"}
Jul 01 21:10:17.813594 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:17.813552    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:10:18.348222 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:18.348190    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" event={"ID":"8b1c2ddd-1999-4917-9046-dc608949bf09","Type":"ContainerStarted","Data":"ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8"}
Jul 01 21:10:18.375078 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:18.375029    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" podStartSLOduration=1.7924836800000001 podStartE2EDuration="6.375013993s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.582530313s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:10:18.373912843 +0000 UTC m=+1451.285247469" watchObservedRunningTime="2026-07-01 21:10:18.375013993 +0000 UTC m=+1451.286348632"
Jul 01 21:10:18.481636 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:18.481608    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:22.481127 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:22.481088    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:10:22.482727 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:22.482694    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 01 21:10:27.813979 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:27.813926    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:10:32.481486 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:32.481437    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 01 21:10:37.813550 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:37.813494    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:10:42.481996 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:42.481961    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 01 21:10:47.813198 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:47.813146    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:10:52.481670 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:52.481626    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 01 21:10:57.813234 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:10:57.813182    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 01 21:11:02.481137 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:02.481093    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 01 21:11:07.621250 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:07.621226    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:11:07.623039 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:07.623012    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:11:07.632274 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:07.632255    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:11:07.633885 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:07.633847    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:11:07.826204 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:07.826175    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:11:07.835712 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:07.835685    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:11:12.481272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:12.481217    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 01 21:11:12.883468 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:12.883435    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"]
Jul 01 21:11:22.481305 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:22.481261    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 01 21:11:27.907186 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:27.907142    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" containerID="cri-o://34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2" gracePeriod=45
Jul 01 21:11:28.859782 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.859754    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:11:28.990288 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.990201    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-dshm\") pod \"87b65003-4119-43ed-9670-23685bacbba9\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") "
Jul 01 21:11:28.990288 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.990256    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-model-cache\") pod \"87b65003-4119-43ed-9670-23685bacbba9\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") "
Jul 01 21:11:28.990288 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.990280    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-home\") pod \"87b65003-4119-43ed-9670-23685bacbba9\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") "
Jul 01 21:11:28.990824 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.990319    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9hnn\" (UniqueName: \"kubernetes.io/projected/87b65003-4119-43ed-9670-23685bacbba9-kube-api-access-m9hnn\") pod \"87b65003-4119-43ed-9670-23685bacbba9\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") "
Jul 01 21:11:28.990824 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.990374    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-kserve-provision-location\") pod \"87b65003-4119-43ed-9670-23685bacbba9\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") "
Jul 01 21:11:28.990824 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.990432    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b65003-4119-43ed-9670-23685bacbba9-tls-certs\") pod \"87b65003-4119-43ed-9670-23685bacbba9\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") "
Jul 01 21:11:28.990824 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.990472    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-tmp-dir\") pod \"87b65003-4119-43ed-9670-23685bacbba9\" (UID: \"87b65003-4119-43ed-9670-23685bacbba9\") "
Jul 01 21:11:28.990824 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.990579    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-model-cache" (OuterVolumeSpecName: "model-cache") pod "87b65003-4119-43ed-9670-23685bacbba9" (UID: "87b65003-4119-43ed-9670-23685bacbba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:11:28.990824 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.990815    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:11:28.991270 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.991244    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-home" (OuterVolumeSpecName: "home") pod "87b65003-4119-43ed-9670-23685bacbba9" (UID: "87b65003-4119-43ed-9670-23685bacbba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:11:28.992786 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.992762    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-dshm" (OuterVolumeSpecName: "dshm") pod "87b65003-4119-43ed-9670-23685bacbba9" (UID: "87b65003-4119-43ed-9670-23685bacbba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:11:28.992927 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.992793    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b65003-4119-43ed-9670-23685bacbba9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87b65003-4119-43ed-9670-23685bacbba9" (UID: "87b65003-4119-43ed-9670-23685bacbba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:11:28.992927 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:28.992817    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b65003-4119-43ed-9670-23685bacbba9-kube-api-access-m9hnn" (OuterVolumeSpecName: "kube-api-access-m9hnn") pod "87b65003-4119-43ed-9670-23685bacbba9" (UID: "87b65003-4119-43ed-9670-23685bacbba9"). InnerVolumeSpecName "kube-api-access-m9hnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:11:29.008790 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.008758    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87b65003-4119-43ed-9670-23685bacbba9" (UID: "87b65003-4119-43ed-9670-23685bacbba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:11:29.053699 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.053665    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87b65003-4119-43ed-9670-23685bacbba9" (UID: "87b65003-4119-43ed-9670-23685bacbba9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:11:29.091646 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.091604    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:11:29.091646 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.091640    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:11:29.091820 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.091653    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9hnn\" (UniqueName: \"kubernetes.io/projected/87b65003-4119-43ed-9670-23685bacbba9-kube-api-access-m9hnn\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:11:29.091820 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.091669    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:11:29.091820 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.091683    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b65003-4119-43ed-9670-23685bacbba9-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:11:29.091820 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.091695    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b65003-4119-43ed-9670-23685bacbba9-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:11:29.709186 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.709150    2573 generic.go:356] "Generic (PLEG): container finished" podID="87b65003-4119-43ed-9670-23685bacbba9" containerID="34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2" exitCode=0
Jul 01 21:11:29.709371 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.709194    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" event={"ID":"87b65003-4119-43ed-9670-23685bacbba9","Type":"ContainerDied","Data":"34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2"}
Jul 01 21:11:29.709371 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.709203    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"
Jul 01 21:11:29.709371 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.709250    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" event={"ID":"87b65003-4119-43ed-9670-23685bacbba9","Type":"ContainerDied","Data":"59ce823cbb154be50c762164c6c7f21d33ab83b1c3c1e5979cd8617098aef8e8"}
Jul 01 21:11:29.709371 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.709279    2573 scope.go:117] "RemoveContainer" containerID="34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2"
Jul 01 21:11:29.719046 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.719026    2573 scope.go:117] "RemoveContainer" containerID="b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183"
Jul 01 21:11:29.726958 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.726941    2573 scope.go:117] "RemoveContainer" containerID="34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2"
Jul 01 21:11:29.727207 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:11:29.727189    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2\": container with ID starting with 34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2 not found: ID does not exist" containerID="34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2"
Jul 01 21:11:29.727257 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.727218    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2"} err="failed to get container status \"34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2\": rpc error: code = NotFound desc = could not find container \"34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2\": container with ID starting with 34ae989b3adf2783cd721d5ef4dfe6925753153d999895e447ced0917a3badf2 not found: ID does not exist"
Jul 01 21:11:29.727257 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.727235    2573 scope.go:117] "RemoveContainer" containerID="b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183"
Jul 01 21:11:29.727503 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:11:29.727487    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183\": container with ID starting with b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183 not found: ID does not exist" containerID="b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183"
Jul 01 21:11:29.727546 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.727509    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183"} err="failed to get container status \"b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183\": rpc error: code = NotFound desc = could not find container \"b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183\": container with ID starting with b49462a5c7d0de320cf1d3c8e4762c88b7b3f2df0275a6c53de4a054087c9183 not found: ID does not exist"
Jul 01 21:11:29.745484 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.745459    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"]
Jul 01 21:11:29.748317 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.748287    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl"]
Jul 01 21:11:29.827557 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:29.827524    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": context deadline exceeded"
Jul 01 21:11:31.591209 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:31.591176    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b65003-4119-43ed-9670-23685bacbba9" path="/var/lib/kubelet/pods/87b65003-4119-43ed-9670-23685bacbba9/volumes"
Jul 01 21:11:32.492043 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:32.492007    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:11:32.500356 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:32.500334    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:11:41.314330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.314284    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"]
Jul 01 21:11:41.314765 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.314749    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="storage-initializer"
Jul 01 21:11:41.314813 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.314769    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="storage-initializer"
Jul 01 21:11:41.314849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.314811    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main"
Jul 01 21:11:41.314849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.314820    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main"
Jul 01 21:11:41.314947 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.314918    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b65003-4119-43ed-9670-23685bacbba9" containerName="main"
Jul 01 21:11:41.317265 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.317248    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.320818 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.320800    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 01 21:11:41.335195 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.335174    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"]
Jul 01 21:11:41.391562 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.391535    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-dshm\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.391696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.391576    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tmp-dir\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.391696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.391593    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgf9q\" (UniqueName: \"kubernetes.io/projected/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kube-api-access-pgf9q\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.391696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.391646    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-model-cache\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.391696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.391682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tls-certs\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.391885 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.391729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-home\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.391885 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.391782    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.492836 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.492802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-home\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493005 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.492900    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493005 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.492951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-dshm\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493005 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.493002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tmp-dir\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493172 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.493025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgf9q\" (UniqueName: \"kubernetes.io/projected/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kube-api-access-pgf9q\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493172 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.493057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-model-cache\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493172 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.493086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tls-certs\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.493253    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-home\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.493314    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493449 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.493433    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-model-cache\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.493604 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.493588    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tmp-dir\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.495413 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.495388    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-dshm\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.495546 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.495531    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tls-certs\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.501523 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.501504    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgf9q\" (UniqueName: \"kubernetes.io/projected/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kube-api-access-pgf9q\") pod \"custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.627899 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.627866    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:41.766876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:41.766832    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"]
Jul 01 21:11:41.769531 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:11:41.769502    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a52cd1_14cc_4cc7_a6d9_f2be5407fa76.slice/crio-832ca9de15a5a0a399bf5099efb654ffafff18d996fddb9d69f775bcdb86abe8 WatchSource:0}: Error finding container 832ca9de15a5a0a399bf5099efb654ffafff18d996fddb9d69f775bcdb86abe8: Status 404 returned error can't find the container with id 832ca9de15a5a0a399bf5099efb654ffafff18d996fddb9d69f775bcdb86abe8
Jul 01 21:11:42.772570 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:42.772514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" event={"ID":"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76","Type":"ContainerStarted","Data":"64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4"}
Jul 01 21:11:42.773270 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:42.772578    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" event={"ID":"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76","Type":"ContainerStarted","Data":"832ca9de15a5a0a399bf5099efb654ffafff18d996fddb9d69f775bcdb86abe8"}
Jul 01 21:11:46.793637 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:46.793606    2573 generic.go:356] "Generic (PLEG): container finished" podID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerID="64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4" exitCode=0
Jul 01 21:11:46.793997 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:46.793679    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" event={"ID":"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76","Type":"ContainerDied","Data":"64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4"}
Jul 01 21:11:47.800174 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:47.800141    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" event={"ID":"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76","Type":"ContainerStarted","Data":"6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff"}
Jul 01 21:11:47.828766 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:47.828711    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podStartSLOduration=2.712676503 podStartE2EDuration="6.828689246s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.116012743s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:11:47.824755462 +0000 UTC m=+1540.736090093" watchObservedRunningTime="2026-07-01 21:11:47.828689246 +0000 UTC m=+1540.740023875"
Jul 01 21:11:48.628498 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:48.628470    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:51.628094 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:51.628063    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:11:51.629616 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:11:51.629582    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:12:01.628605 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:12:01.628561    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:12:11.629124 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:12:11.629022    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:12:21.628352 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:12:21.628299    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:12:31.629176 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:12:31.629140    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:12:41.628296 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:12:41.628251    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:12:51.628942 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:12:51.628897    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:13:01.628553 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:01.628472    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:13:11.629204 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:11.629152    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:13:21.629306 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:21.629266    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:13:31.629166 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:31.629123    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:13:37.642364 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:37.642306    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"]
Jul 01 21:13:41.637997 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:41.637964    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:13:41.645713 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:41.645690    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:13:46.119935 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:46.119903    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"]
Jul 01 21:13:52.665216 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:52.665159    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main" containerID="cri-o://ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8" gracePeriod=45
Jul 01 21:13:53.317683 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.317658    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:13:53.389838 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.389768    2573 generic.go:356] "Generic (PLEG): container finished" podID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerID="ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8" exitCode=0
Jul 01 21:13:53.389838 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.389824    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"
Jul 01 21:13:53.390043 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.389850    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" event={"ID":"8b1c2ddd-1999-4917-9046-dc608949bf09","Type":"ContainerDied","Data":"ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8"}
Jul 01 21:13:53.390043 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.389931    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj" event={"ID":"8b1c2ddd-1999-4917-9046-dc608949bf09","Type":"ContainerDied","Data":"aece49db9eb73661a27d056874c29cc01a3a85408864cc7861835a033b33189b"}
Jul 01 21:13:53.390043 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.389955    2573 scope.go:117] "RemoveContainer" containerID="ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8"
Jul 01 21:13:53.400528 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.400510    2573 scope.go:117] "RemoveContainer" containerID="5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01"
Jul 01 21:13:53.408265 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408249    2573 scope.go:117] "RemoveContainer" containerID="ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8"
Jul 01 21:13:53.408495 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:13:53.408476    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8\": container with ID starting with ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8 not found: ID does not exist" containerID="ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8"
Jul 01 21:13:53.408544 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408509    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8"} err="failed to get container status \"ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8\": rpc error: code = NotFound desc = could not find container \"ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8\": container with ID starting with ebe9cd98439c78d82e2d898d02a4369f42f4e99d01ffefc340d5784d9937dcf8 not found: ID does not exist"
Jul 01 21:13:53.408544 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408531    2573 scope.go:117] "RemoveContainer" containerID="5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01"
Jul 01 21:13:53.408628 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408571    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-dshm\") pod \"8b1c2ddd-1999-4917-9046-dc608949bf09\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") "
Jul 01 21:13:53.408688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408631    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-model-cache\") pod \"8b1c2ddd-1999-4917-9046-dc608949bf09\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") "
Jul 01 21:13:53.408688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408653    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c2ddd-1999-4917-9046-dc608949bf09-cabundle-cert\") pod \"8b1c2ddd-1999-4917-9046-dc608949bf09\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") "
Jul 01 21:13:53.408788 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408695    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-home\") pod \"8b1c2ddd-1999-4917-9046-dc608949bf09\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") "
Jul 01 21:13:53.408788 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408736    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-tmp-dir\") pod \"8b1c2ddd-1999-4917-9046-dc608949bf09\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") "
Jul 01 21:13:53.408912 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:13:53.408819    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01\": container with ID starting with 5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01 not found: ID does not exist" containerID="5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01"
Jul 01 21:13:53.408912 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408867    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01"} err="failed to get container status \"5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01\": rpc error: code = NotFound desc = could not find container \"5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01\": container with ID starting with 5f43feb54d917754011ba12ea1f00d9a30400265113ce05e9d912aa2ee541e01 not found: ID does not exist"
Jul 01 21:13:53.409024 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408889    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b1c2ddd-1999-4917-9046-dc608949bf09" (UID: "8b1c2ddd-1999-4917-9046-dc608949bf09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:13:53.409024 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408905    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-kserve-provision-location\") pod \"8b1c2ddd-1999-4917-9046-dc608949bf09\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") "
Jul 01 21:13:53.409024 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.408985    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1c2ddd-1999-4917-9046-dc608949bf09-tls-certs\") pod \"8b1c2ddd-1999-4917-9046-dc608949bf09\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") "
Jul 01 21:13:53.409210 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.409025    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx2sz\" (UniqueName: \"kubernetes.io/projected/8b1c2ddd-1999-4917-9046-dc608949bf09-kube-api-access-zx2sz\") pod \"8b1c2ddd-1999-4917-9046-dc608949bf09\" (UID: \"8b1c2ddd-1999-4917-9046-dc608949bf09\") "
Jul 01 21:13:53.409210 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.409076    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1c2ddd-1999-4917-9046-dc608949bf09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b1c2ddd-1999-4917-9046-dc608949bf09" (UID: "8b1c2ddd-1999-4917-9046-dc608949bf09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:13:53.409210 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.409165    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b1c2ddd-1999-4917-9046-dc608949bf09" (UID: "8b1c2ddd-1999-4917-9046-dc608949bf09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:13:53.409362 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.409254    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:13:53.409362 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.409273    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c2ddd-1999-4917-9046-dc608949bf09-cabundle-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:13:53.409362 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.409285    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:13:53.409362 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.409312    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-home" (OuterVolumeSpecName: "home") pod "8b1c2ddd-1999-4917-9046-dc608949bf09" (UID: "8b1c2ddd-1999-4917-9046-dc608949bf09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:13:53.411033 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.411003    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1c2ddd-1999-4917-9046-dc608949bf09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b1c2ddd-1999-4917-9046-dc608949bf09" (UID: "8b1c2ddd-1999-4917-9046-dc608949bf09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:13:53.411121 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.411044    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1c2ddd-1999-4917-9046-dc608949bf09-kube-api-access-zx2sz" (OuterVolumeSpecName: "kube-api-access-zx2sz") pod "8b1c2ddd-1999-4917-9046-dc608949bf09" (UID: "8b1c2ddd-1999-4917-9046-dc608949bf09"). InnerVolumeSpecName "kube-api-access-zx2sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:13:53.411121 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.411061    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-dshm" (OuterVolumeSpecName: "dshm") pod "8b1c2ddd-1999-4917-9046-dc608949bf09" (UID: "8b1c2ddd-1999-4917-9046-dc608949bf09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:13:53.471086 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.471056    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b1c2ddd-1999-4917-9046-dc608949bf09" (UID: "8b1c2ddd-1999-4917-9046-dc608949bf09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:13:53.509848 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.509827    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:13:53.509940 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.509848    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b1c2ddd-1999-4917-9046-dc608949bf09-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:13:53.509940 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.509875    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx2sz\" (UniqueName: \"kubernetes.io/projected/8b1c2ddd-1999-4917-9046-dc608949bf09-kube-api-access-zx2sz\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:13:53.509940 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.509883    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:13:53.509940 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.509892    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b1c2ddd-1999-4917-9046-dc608949bf09-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:13:53.707026 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.706953    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"]
Jul 01 21:13:53.713383 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:53.713358    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj"]
Jul 01 21:13:55.591168 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:13:55.591131    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" path="/var/lib/kubelet/pods/8b1c2ddd-1999-4917-9046-dc608949bf09/volumes"
Jul 01 21:14:01.146954 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.146891    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" containerID="cri-o://6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff" gracePeriod=45
Jul 01 21:14:01.639566 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.639527    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 01 21:14:01.894694 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.894629    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:14:01.970198 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.970169    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-model-cache\") pod \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") "
Jul 01 21:14:01.970332 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.970241    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kserve-provision-location\") pod \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") "
Jul 01 21:14:01.970377 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.970326    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-home\") pod \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") "
Jul 01 21:14:01.970434 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.970375    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgf9q\" (UniqueName: \"kubernetes.io/projected/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kube-api-access-pgf9q\") pod \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") "
Jul 01 21:14:01.970434 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.970414    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tmp-dir\") pod \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") "
Jul 01 21:14:01.970554 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.970412    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-model-cache" (OuterVolumeSpecName: "model-cache") pod "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" (UID: "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:14:01.970554 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.970464    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-dshm\") pod \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") "
Jul 01 21:14:01.970554 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.970499    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tls-certs\") pod \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\" (UID: \"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76\") "
Jul 01 21:14:01.970753 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.970732    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:14:01.971041 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.971016    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-home" (OuterVolumeSpecName: "home") pod "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" (UID: "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:14:01.973007 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.972972    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-dshm" (OuterVolumeSpecName: "dshm") pod "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" (UID: "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:14:01.973139 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.973111    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kube-api-access-pgf9q" (OuterVolumeSpecName: "kube-api-access-pgf9q") pod "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" (UID: "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76"). InnerVolumeSpecName "kube-api-access-pgf9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:14:01.973239 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.973225    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" (UID: "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:14:01.983792 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:01.983767    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" (UID: "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:14:02.032981 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.032951    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" (UID: "20a52cd1-14cc-4cc7-a6d9-f2be5407fa76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:14:02.071738 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.071716    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgf9q\" (UniqueName: \"kubernetes.io/projected/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kube-api-access-pgf9q\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:14:02.071823 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.071738    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:14:02.071823 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.071749    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:14:02.071823 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.071757    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:14:02.071823 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.071764    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:14:02.071823 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.071772    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:14:02.431776 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.431745    2573 generic.go:356] "Generic (PLEG): container finished" podID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerID="6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff" exitCode=0
Jul 01 21:14:02.432175 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.431804    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"
Jul 01 21:14:02.432175 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.431835    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" event={"ID":"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76","Type":"ContainerDied","Data":"6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff"}
Jul 01 21:14:02.432175 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.431910    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv" event={"ID":"20a52cd1-14cc-4cc7-a6d9-f2be5407fa76","Type":"ContainerDied","Data":"832ca9de15a5a0a399bf5099efb654ffafff18d996fddb9d69f775bcdb86abe8"}
Jul 01 21:14:02.432175 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.431942    2573 scope.go:117] "RemoveContainer" containerID="6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff"
Jul 01 21:14:02.441004 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.440989    2573 scope.go:117] "RemoveContainer" containerID="64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4"
Jul 01 21:14:02.448623 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.448599    2573 scope.go:117] "RemoveContainer" containerID="6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff"
Jul 01 21:14:02.448850 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:14:02.448832    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff\": container with ID starting with 6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff not found: ID does not exist" containerID="6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff"
Jul 01 21:14:02.448912 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.448872    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff"} err="failed to get container status \"6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff\": rpc error: code = NotFound desc = could not find container \"6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff\": container with ID starting with 6c5266eca282f2594e100f23c34e748ef8d18fa4f0885b8c36f41fc5b6e75dff not found: ID does not exist"
Jul 01 21:14:02.448912 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.448890    2573 scope.go:117] "RemoveContainer" containerID="64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4"
Jul 01 21:14:02.449159 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:14:02.449143    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4\": container with ID starting with 64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4 not found: ID does not exist" containerID="64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4"
Jul 01 21:14:02.449209 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.449164    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4"} err="failed to get container status \"64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4\": rpc error: code = NotFound desc = could not find container \"64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4\": container with ID starting with 64c447e21bb1fda865506a0d8726246e21ffb804f672bc346c5eb9e1026aa3e4 not found: ID does not exist"
Jul 01 21:14:02.453695 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.453673    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"]
Jul 01 21:14:02.460334 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:02.460314    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv"]
Jul 01 21:14:03.590793 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:03.590761    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" path="/var/lib/kubelet/pods/20a52cd1-14cc-4cc7-a6d9-f2be5407fa76/volumes"
Jul 01 21:14:21.779914 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.779879    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"]
Jul 01 21:14:21.780344 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780316    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main"
Jul 01 21:14:21.780344 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780331    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main"
Jul 01 21:14:21.780418 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780353    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="storage-initializer"
Jul 01 21:14:21.780418 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780362    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="storage-initializer"
Jul 01 21:14:21.780418 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780381    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main"
Jul 01 21:14:21.780418 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780389    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main"
Jul 01 21:14:21.780418 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780405    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="storage-initializer"
Jul 01 21:14:21.780418 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780413    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="storage-initializer"
Jul 01 21:14:21.780610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780486    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a52cd1-14cc-4cc7-a6d9-f2be5407fa76" containerName="main"
Jul 01 21:14:21.780610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.780498    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1c2ddd-1999-4917-9046-dc608949bf09" containerName="main"
Jul 01 21:14:21.785522 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.785499    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.788686 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.788665    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:14:21.788799 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.788682    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 21:14:21.788799 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.788756    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-76tz7\""
Jul 01 21:14:21.789001 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.788938    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 21:14:21.789001 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.788964    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 01 21:14:21.795590 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.795567    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"]
Jul 01 21:14:21.813981 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.813948    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-home\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.814085 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.813987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lxs\" (UniqueName: \"kubernetes.io/projected/bdad64c2-c804-49cc-bea7-74379d67af1a-kube-api-access-n4lxs\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.814141 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.814009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-model-cache\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.814197 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.814162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-tmp-dir\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.814243 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.814216    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-dshm\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.814283 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.814246    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-kserve-provision-location\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.814322 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.814308    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdad64c2-c804-49cc-bea7-74379d67af1a-tls-certs\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915090 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdad64c2-c804-49cc-bea7-74379d67af1a-tls-certs\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915189 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-home\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915189 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915128    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lxs\" (UniqueName: \"kubernetes.io/projected/bdad64c2-c804-49cc-bea7-74379d67af1a-kube-api-access-n4lxs\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915189 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-model-cache\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915189 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-tmp-dir\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915358 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-dshm\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915358 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915244    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-kserve-provision-location\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915529 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915508    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-home\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915602 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915551    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-model-cache\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915602 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915580    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-kserve-provision-location\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.915703 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.915639    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-tmp-dir\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.917338 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.917317    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-dshm\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.917604 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.917587    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdad64c2-c804-49cc-bea7-74379d67af1a-tls-certs\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:21.923491 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:21.923469    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lxs\" (UniqueName: \"kubernetes.io/projected/bdad64c2-c804-49cc-bea7-74379d67af1a-kube-api-access-n4lxs\") pod \"router-with-refs-test-kserve-7948b44cf6-7jnwf\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:22.018519 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.018488    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"]
Jul 01 21:14:22.022119 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.022100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.025476 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.025460    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-56999\""
Jul 01 21:14:22.032049 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.031999    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"]
Jul 01 21:14:22.097569 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.097547    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:22.117391 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.117365    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.117496 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.117408    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.117496 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.117459    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4f9b\" (UniqueName: \"kubernetes.io/projected/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kube-api-access-h4f9b\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.117496 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.117486    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.218328 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.218293    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4f9b\" (UniqueName: \"kubernetes.io/projected/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kube-api-access-h4f9b\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.218454 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.218339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.218454 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.218374    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.218454 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.218404    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.218728 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.218713    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.218793 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.218773    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.220712 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.220693    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.230279 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.230256    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4f9b\" (UniqueName: \"kubernetes.io/projected/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kube-api-access-h4f9b\") pod \"router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.254275 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.254253    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"]
Jul 01 21:14:22.258342 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:14:22.258314    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdad64c2_c804_49cc_bea7_74379d67af1a.slice/crio-a95d2bc0c2c7b0e39f0418de5e4f3aed07e74df9c445732bd824c6dccda67776 WatchSource:0}: Error finding container a95d2bc0c2c7b0e39f0418de5e4f3aed07e74df9c445732bd824c6dccda67776: Status 404 returned error can't find the container with id a95d2bc0c2c7b0e39f0418de5e4f3aed07e74df9c445732bd824c6dccda67776
Jul 01 21:14:22.331584 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.331564    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:22.462725 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.462702    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"]
Jul 01 21:14:22.464075 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:14:22.464049    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc7f1f1_aa8e_40ca_bc2e_a5297658de7d.slice/crio-5f895148411d78a6f432af6c4b2a071f426328bcc532eb3ba9a512f8c85a33e0 WatchSource:0}: Error finding container 5f895148411d78a6f432af6c4b2a071f426328bcc532eb3ba9a512f8c85a33e0: Status 404 returned error can't find the container with id 5f895148411d78a6f432af6c4b2a071f426328bcc532eb3ba9a512f8c85a33e0
Jul 01 21:14:22.519107 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.519078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" event={"ID":"bdad64c2-c804-49cc-bea7-74379d67af1a","Type":"ContainerStarted","Data":"32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c"}
Jul 01 21:14:22.519227 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.519135    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" event={"ID":"bdad64c2-c804-49cc-bea7-74379d67af1a","Type":"ContainerStarted","Data":"a95d2bc0c2c7b0e39f0418de5e4f3aed07e74df9c445732bd824c6dccda67776"}
Jul 01 21:14:22.520632 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:22.520604    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s" event={"ID":"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d","Type":"ContainerStarted","Data":"5f895148411d78a6f432af6c4b2a071f426328bcc532eb3ba9a512f8c85a33e0"}
Jul 01 21:14:23.527648 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:23.527550    2573 generic.go:356] "Generic (PLEG): container finished" podID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" containerID="4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95" exitCode=0
Jul 01 21:14:23.527648 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:23.527621    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s" event={"ID":"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d","Type":"ContainerDied","Data":"4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95"}
Jul 01 21:14:24.535527 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:24.535491    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s" event={"ID":"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d","Type":"ContainerStarted","Data":"88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d"}
Jul 01 21:14:24.535950 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:24.535627    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:24.557542 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:24.557487    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s" podStartSLOduration=1.882028986 podStartE2EDuration="2.557473742s" totalImagesPullingTime="0s" totalInitContainerRuntime="675.444756ms" isStatefulPod=false podCreationTimestamp="2026-07-01 21:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:14:24.555873508 +0000 UTC m=+1697.467208134" watchObservedRunningTime="2026-07-01 21:14:24.557473742 +0000 UTC m=+1697.468808386"
Jul 01 21:14:26.546348 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:26.546315    2573 generic.go:356] "Generic (PLEG): container finished" podID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerID="32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c" exitCode=0
Jul 01 21:14:26.546720 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:26.546387    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" event={"ID":"bdad64c2-c804-49cc-bea7-74379d67af1a","Type":"ContainerDied","Data":"32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c"}
Jul 01 21:14:27.552823 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:27.552789    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" event={"ID":"bdad64c2-c804-49cc-bea7-74379d67af1a","Type":"ContainerStarted","Data":"b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1"}
Jul 01 21:14:27.578001 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:27.577948    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podStartSLOduration=3.065637824 podStartE2EDuration="6.577930002s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.512292178s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:14:27.577442582 +0000 UTC m=+1700.488777198" watchObservedRunningTime="2026-07-01 21:14:27.577930002 +0000 UTC m=+1700.489264662"
Jul 01 21:14:28.098460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:28.098420    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:32.097978 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:32.097887    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:14:32.099352 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:32.099323    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:14:42.098985 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:42.098935    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:14:52.097948 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.097909    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:14:52.186914 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.186883    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"]
Jul 01 21:14:52.189163 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.189147    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.192083 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.192058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 01 21:14:52.202250 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.202224    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"]
Jul 01 21:14:52.273215 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.273188    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.273378 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.273224    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrzb\" (UniqueName: \"kubernetes.io/projected/e83a902f-c8b7-4df3-a590-c4a5363287b9-kube-api-access-znrzb\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.273378 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.273250    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.273378 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.273270    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83a902f-c8b7-4df3-a590-c4a5363287b9-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.273378 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.273289    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.273378 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.273362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.374288 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.374198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.374288 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.374238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrzb\" (UniqueName: \"kubernetes.io/projected/e83a902f-c8b7-4df3-a590-c4a5363287b9-kube-api-access-znrzb\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.374288 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.374272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.374572 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.374301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83a902f-c8b7-4df3-a590-c4a5363287b9-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.374572 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.374328    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.374572 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.374375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.374754 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.374730    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.374813 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.374775    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.374813 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.374806    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.376542 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.376514    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.376749 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.376733    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83a902f-c8b7-4df3-a590-c4a5363287b9-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:52.384053 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:52.384030    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrzb\" (UniqueName: \"kubernetes.io/projected/e83a902f-c8b7-4df3-a590-c4a5363287b9-kube-api-access-znrzb\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:54.693740 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:54.693699    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:54.701578 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:54.701551    2573 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:54.795115 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:54.795078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:54.795276 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:54.795214    2573 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:54.799352 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:54.799325    2573 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:54.894104 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:54.894068    2573 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a7984d75592a60f75acc48715216990326d59cb841472d1b5739e644ba9a7d8d/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:54.910449 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:54.910423    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:55.198505 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:55.198470    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:14:55.344371 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:55.344346    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"]
Jul 01 21:14:55.346096 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:14:55.346056    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83a902f_c8b7_4df3_a590_c4a5363287b9.slice/crio-936cdc5c5f291f2f67bbcf3eae4752ff4618b718bd782c9173e426226c0d0620 WatchSource:0}: Error finding container 936cdc5c5f291f2f67bbcf3eae4752ff4618b718bd782c9173e426226c0d0620: Status 404 returned error can't find the container with id 936cdc5c5f291f2f67bbcf3eae4752ff4618b718bd782c9173e426226c0d0620
Jul 01 21:14:55.543804 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:55.543707    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:14:55.687142 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:55.687107    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" event={"ID":"e83a902f-c8b7-4df3-a590-c4a5363287b9","Type":"ContainerStarted","Data":"c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9"}
Jul 01 21:14:55.687313 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:55.687175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" event={"ID":"e83a902f-c8b7-4df3-a590-c4a5363287b9","Type":"ContainerStarted","Data":"936cdc5c5f291f2f67bbcf3eae4752ff4618b718bd782c9173e426226c0d0620"}
Jul 01 21:14:55.707651 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:55.707601    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podStartSLOduration=3.707581382 podStartE2EDuration="3.707581382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:14:55.706663797 +0000 UTC m=+1728.617998423" watchObservedRunningTime="2026-07-01 21:14:55.707581382 +0000 UTC m=+1728.618916011"
Jul 01 21:14:56.199330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:14:56.199299    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:15:02.098537 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:02.098491    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:15:05.199257 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:05.199211    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:15:05.200702 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:05.200670    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:15:12.098643 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:12.098574    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:15:15.199503 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:15.199442    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:15:22.098653 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:22.098607    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:15:25.199485 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:25.199425    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:15:32.098137 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:32.098084    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:15:35.199677 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:35.199624    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:15:42.098935 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:42.098887    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:15:45.199057 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:45.199002    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:15:52.098215 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:52.098160    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:15:55.199222 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:15:55.199178    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:16:02.099033 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:02.098932    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:16:05.199124 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:05.199086    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:16:07.654038 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:07.653994    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:16:07.657332 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:07.657312    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:16:07.661593 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:07.661572    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:16:07.664451 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:07.664433    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:16:12.098647 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:12.098599    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:16:15.199365 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:15.199322    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:16:22.098693 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:22.098623    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 01 21:16:25.199201 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:25.199161    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:16:32.108801 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:32.108757    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:16:32.116635 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:32.116604    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:16:35.199019 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:35.198975    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:16:40.307183 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.307140    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"]
Jul 01 21:16:40.317581 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:16:40.317538    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 21:16:40.317581 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:16:40.317581 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:16:40.317581 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d" cmd=["/bin/sleep","15"]
Jul 01 21:16:40.317926 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.317590    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s" message="" err=<
Jul 01 21:16:40.317926 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:16:40.317926 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:16:40.317926 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 21:16:40.317926 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:16:40.317606    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 21:16:40.317926 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:16:40.317926 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:16:40.317926 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s" podUID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" containerName="main" containerID="cri-o://88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d"
Jul 01 21:16:40.317926 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.317644    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s" podUID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" containerName="main" containerID="cri-o://88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d" gracePeriod=60
Jul 01 21:16:40.320897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.320872    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"]
Jul 01 21:16:40.567994 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.567973    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:16:40.623519 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.623488    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4f9b\" (UniqueName: \"kubernetes.io/projected/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kube-api-access-h4f9b\") pod \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") "
Jul 01 21:16:40.623665 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.623570    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tokenizer-uds\") pod \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") "
Jul 01 21:16:40.623665 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.623623    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tls-certs\") pod \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") "
Jul 01 21:16:40.623665 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.623643    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kserve-provision-location\") pod \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\" (UID: \"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d\") "
Jul 01 21:16:40.623891 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.623847    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" (UID: "9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:16:40.624543 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.624512    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" (UID: "9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:16:40.625773 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.625745    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" (UID: "9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:16:40.625773 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.625752    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kube-api-access-h4f9b" (OuterVolumeSpecName: "kube-api-access-h4f9b") pod "9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" (UID: "9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d"). InnerVolumeSpecName "kube-api-access-h4f9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:16:40.725332 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.725301    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:40.725462 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.725337    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:40.725462 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.725352    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:40.725462 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:40.725367    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4f9b\" (UniqueName: \"kubernetes.io/projected/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d-kube-api-access-h4f9b\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:41.230987 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.230955    2573 generic.go:356] "Generic (PLEG): container finished" podID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" containerID="88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d" exitCode=0
Jul 01 21:16:41.231159 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.231005    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"
Jul 01 21:16:41.231159 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.231036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s" event={"ID":"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d","Type":"ContainerDied","Data":"88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d"}
Jul 01 21:16:41.231159 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.231091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s" event={"ID":"9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d","Type":"ContainerDied","Data":"5f895148411d78a6f432af6c4b2a071f426328bcc532eb3ba9a512f8c85a33e0"}
Jul 01 21:16:41.231159 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.231120    2573 scope.go:117] "RemoveContainer" containerID="88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d"
Jul 01 21:16:41.240601 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.240578    2573 scope.go:117] "RemoveContainer" containerID="4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95"
Jul 01 21:16:41.248164 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.248136    2573 scope.go:117] "RemoveContainer" containerID="88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d"
Jul 01 21:16:41.248418 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:16:41.248401    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d\": container with ID starting with 88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d not found: ID does not exist" containerID="88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d"
Jul 01 21:16:41.248473 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.248427    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d"} err="failed to get container status \"88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d\": rpc error: code = NotFound desc = could not find container \"88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d\": container with ID starting with 88cd4b27a8ab6950b86913d2a1174e75b30e422816b74e127b8cad73c7c90b9d not found: ID does not exist"
Jul 01 21:16:41.248473 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.248444    2573 scope.go:117] "RemoveContainer" containerID="4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95"
Jul 01 21:16:41.248685 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:16:41.248664    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95\": container with ID starting with 4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95 not found: ID does not exist" containerID="4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95"
Jul 01 21:16:41.248765 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.248689    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95"} err="failed to get container status \"4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95\": rpc error: code = NotFound desc = could not find container \"4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95\": container with ID starting with 4433e8859a758f6a64260bf2ec563b910d582d9587656b16a125dc737326ba95 not found: ID does not exist"
Jul 01 21:16:41.260886 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.260850    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"]
Jul 01 21:16:41.263231 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.263213    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s"]
Jul 01 21:16:41.593721 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:41.593687    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" path="/var/lib/kubelet/pods/9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d/volumes"
Jul 01 21:16:45.199071 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:45.199024    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:16:55.199390 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:55.199348    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 01 21:16:55.345809 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:55.345771    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main" containerID="cri-o://b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1" gracePeriod=45
Jul 01 21:16:56.094742 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.094717    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:16:56.269213 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.269134    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-kserve-provision-location\") pod \"bdad64c2-c804-49cc-bea7-74379d67af1a\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") "
Jul 01 21:16:56.269213 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.269174    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-tmp-dir\") pod \"bdad64c2-c804-49cc-bea7-74379d67af1a\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") "
Jul 01 21:16:56.269213 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.269200    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-home\") pod \"bdad64c2-c804-49cc-bea7-74379d67af1a\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") "
Jul 01 21:16:56.269748 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.269230    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdad64c2-c804-49cc-bea7-74379d67af1a-tls-certs\") pod \"bdad64c2-c804-49cc-bea7-74379d67af1a\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") "
Jul 01 21:16:56.269748 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.269263    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-dshm\") pod \"bdad64c2-c804-49cc-bea7-74379d67af1a\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") "
Jul 01 21:16:56.269748 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.269312    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-model-cache\") pod \"bdad64c2-c804-49cc-bea7-74379d67af1a\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") "
Jul 01 21:16:56.269748 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.269340    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4lxs\" (UniqueName: \"kubernetes.io/projected/bdad64c2-c804-49cc-bea7-74379d67af1a-kube-api-access-n4lxs\") pod \"bdad64c2-c804-49cc-bea7-74379d67af1a\" (UID: \"bdad64c2-c804-49cc-bea7-74379d67af1a\") "
Jul 01 21:16:56.269748 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.269672    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdad64c2-c804-49cc-bea7-74379d67af1a" (UID: "bdad64c2-c804-49cc-bea7-74379d67af1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:16:56.270009 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.269914    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-home" (OuterVolumeSpecName: "home") pod "bdad64c2-c804-49cc-bea7-74379d67af1a" (UID: "bdad64c2-c804-49cc-bea7-74379d67af1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:16:56.271923 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.271888    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdad64c2-c804-49cc-bea7-74379d67af1a-kube-api-access-n4lxs" (OuterVolumeSpecName: "kube-api-access-n4lxs") pod "bdad64c2-c804-49cc-bea7-74379d67af1a" (UID: "bdad64c2-c804-49cc-bea7-74379d67af1a"). InnerVolumeSpecName "kube-api-access-n4lxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:16:56.272047 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.271984    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdad64c2-c804-49cc-bea7-74379d67af1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdad64c2-c804-49cc-bea7-74379d67af1a" (UID: "bdad64c2-c804-49cc-bea7-74379d67af1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:16:56.272108 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.272067    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-dshm" (OuterVolumeSpecName: "dshm") pod "bdad64c2-c804-49cc-bea7-74379d67af1a" (UID: "bdad64c2-c804-49cc-bea7-74379d67af1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:16:56.281976 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.281953    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdad64c2-c804-49cc-bea7-74379d67af1a" (UID: "bdad64c2-c804-49cc-bea7-74379d67af1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:16:56.303023 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.302995    2573 generic.go:356] "Generic (PLEG): container finished" podID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerID="b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1" exitCode=0
Jul 01 21:16:56.303115 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.303052    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"
Jul 01 21:16:56.303115 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.303072    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" event={"ID":"bdad64c2-c804-49cc-bea7-74379d67af1a","Type":"ContainerDied","Data":"b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1"}
Jul 01 21:16:56.303234 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.303128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf" event={"ID":"bdad64c2-c804-49cc-bea7-74379d67af1a","Type":"ContainerDied","Data":"a95d2bc0c2c7b0e39f0418de5e4f3aed07e74df9c445732bd824c6dccda67776"}
Jul 01 21:16:56.303234 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.303150    2573 scope.go:117] "RemoveContainer" containerID="b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1"
Jul 01 21:16:56.311171 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.311154    2573 scope.go:117] "RemoveContainer" containerID="32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c"
Jul 01 21:16:56.320457 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.320437    2573 scope.go:117] "RemoveContainer" containerID="b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1"
Jul 01 21:16:56.320694 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:16:56.320676    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1\": container with ID starting with b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1 not found: ID does not exist" containerID="b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1"
Jul 01 21:16:56.320769 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.320705    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1"} err="failed to get container status \"b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1\": rpc error: code = NotFound desc = could not find container \"b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1\": container with ID starting with b7e1844ea7f564ebbed15531485e7acdfc2a5c52cb84ba7566bae665803f28f1 not found: ID does not exist"
Jul 01 21:16:56.320769 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.320727    2573 scope.go:117] "RemoveContainer" containerID="32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c"
Jul 01 21:16:56.321023 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:16:56.321005    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c\": container with ID starting with 32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c not found: ID does not exist" containerID="32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c"
Jul 01 21:16:56.321073 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.321030    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c"} err="failed to get container status \"32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c\": rpc error: code = NotFound desc = could not find container \"32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c\": container with ID starting with 32c581e8b44ab1d2e23721e407449c23bbc5b12b7670867d760eff27b180459c not found: ID does not exist"
Jul 01 21:16:56.335863 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.335828    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdad64c2-c804-49cc-bea7-74379d67af1a" (UID: "bdad64c2-c804-49cc-bea7-74379d67af1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:16:56.370297 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.370276    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.370297 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.370296    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4lxs\" (UniqueName: \"kubernetes.io/projected/bdad64c2-c804-49cc-bea7-74379d67af1a-kube-api-access-n4lxs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.370392 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.370306    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.370392 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.370315    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.370392 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.370324    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.370392 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.370331    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdad64c2-c804-49cc-bea7-74379d67af1a-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.370392 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.370338    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdad64c2-c804-49cc-bea7-74379d67af1a-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.630977 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.630942    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"]
Jul 01 21:16:56.636579 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:56.636554    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf"]
Jul 01 21:16:57.591205 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:16:57.591153    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" path="/var/lib/kubelet/pods/bdad64c2-c804-49cc-bea7-74379d67af1a/volumes"
Jul 01 21:17:05.209459 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:05.209427    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:17:05.217047 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:05.217028    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:17:06.453955 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.453923    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"]
Jul 01 21:17:06.454395 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454382    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="storage-initializer"
Jul 01 21:17:06.454470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454402    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="storage-initializer"
Jul 01 21:17:06.454470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454415    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" containerName="storage-initializer"
Jul 01 21:17:06.454470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454423    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" containerName="storage-initializer"
Jul 01 21:17:06.454470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454441    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" containerName="main"
Jul 01 21:17:06.454470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454450    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" containerName="main"
Jul 01 21:17:06.454470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454461    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main"
Jul 01 21:17:06.454470 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454469    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main"
Jul 01 21:17:06.454843 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454544    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdad64c2-c804-49cc-bea7-74379d67af1a" containerName="main"
Jul 01 21:17:06.454843 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.454559    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc7f1f1-aa8e-40ca-bc2e-a5297658de7d" containerName="main"
Jul 01 21:17:06.459250 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.459228    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.463523 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.463505    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 01 21:17:06.468652 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.468630    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:17:06.468894 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.468879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-8sb9l\""
Jul 01 21:17:06.488407 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.488385    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"]
Jul 01 21:17:06.499355 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.499330    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"]
Jul 01 21:17:06.502683 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.502665    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.523081 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.523062    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"]
Jul 01 21:17:06.552714 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.552677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.552851 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.552732    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.552851 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.552770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.552851 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.552812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.553019 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.552845    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.553019 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.552938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvls8\" (UniqueName: \"kubernetes.io/projected/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kube-api-access-pvls8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.553019 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.552957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.653407 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.653574 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.653574 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.653717 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.653717 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.653717 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653675    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.653717 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653699    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.653969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.653969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653742    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9979355a-45a2-4171-9164-8afa824810e7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.653969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653768    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.653969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653798    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxnb9\" (UniqueName: \"kubernetes.io/projected/9979355a-45a2-4171-9164-8afa824810e7-kube-api-access-xxnb9\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.653969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653809    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.653969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653887    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.653969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvls8\" (UniqueName: \"kubernetes.io/projected/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kube-api-access-pvls8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.653969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.654372 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.653986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.654372 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.654064    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.654372 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.654177    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.655807 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.655785    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.656218 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.656198    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.667968 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.667948    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvls8\" (UniqueName: \"kubernetes.io/projected/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kube-api-access-pvls8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.755143 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755082    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755143 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755116    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755301 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755301 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9979355a-45a2-4171-9164-8afa824810e7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755301 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755301 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxnb9\" (UniqueName: \"kubernetes.io/projected/9979355a-45a2-4171-9164-8afa824810e7-kube-api-access-xxnb9\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755512 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755512 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755462    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755638 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755619    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755744 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755722    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.755807 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.755755    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.757054 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.757035    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.757375 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.757359    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9979355a-45a2-4171-9164-8afa824810e7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.765940 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.765919    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxnb9\" (UniqueName: \"kubernetes.io/projected/9979355a-45a2-4171-9164-8afa824810e7-kube-api-access-xxnb9\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.767574 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.767552    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:06.812231 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.812197    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:06.926844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.926814    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"]
Jul 01 21:17:06.928671 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:17:06.928638    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003dc2a1_a2b1_4659_99e8_49b2e73b50ca.slice/crio-01ff1cd4d342bfac59b7c4ce82ae0a61a9f71973bcfdcbb41f16a7e7fb075325 WatchSource:0}: Error finding container 01ff1cd4d342bfac59b7c4ce82ae0a61a9f71973bcfdcbb41f16a7e7fb075325: Status 404 returned error can't find the container with id 01ff1cd4d342bfac59b7c4ce82ae0a61a9f71973bcfdcbb41f16a7e7fb075325
Jul 01 21:17:06.931501 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.931483    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 21:17:06.965644 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:06.965623    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"]
Jul 01 21:17:06.968162 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:17:06.968138    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9979355a_45a2_4171_9164_8afa824810e7.slice/crio-a5c718d987e683082dba3c2986b97412a98bc38445af4b3e6f590ad94b73a0f9 WatchSource:0}: Error finding container a5c718d987e683082dba3c2986b97412a98bc38445af4b3e6f590ad94b73a0f9: Status 404 returned error can't find the container with id a5c718d987e683082dba3c2986b97412a98bc38445af4b3e6f590ad94b73a0f9
Jul 01 21:17:07.354533 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:07.354494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" event={"ID":"003dc2a1-a2b1-4659-99e8-49b2e73b50ca","Type":"ContainerStarted","Data":"01ff1cd4d342bfac59b7c4ce82ae0a61a9f71973bcfdcbb41f16a7e7fb075325"}
Jul 01 21:17:07.356113 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:07.356079    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" event={"ID":"9979355a-45a2-4171-9164-8afa824810e7","Type":"ContainerStarted","Data":"477c85e8444bd0b64c90ae541d0ecd5626636165763504ea06ed6497a75264d8"}
Jul 01 21:17:07.356258 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:07.356130    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" event={"ID":"9979355a-45a2-4171-9164-8afa824810e7","Type":"ContainerStarted","Data":"a5c718d987e683082dba3c2986b97412a98bc38445af4b3e6f590ad94b73a0f9"}
Jul 01 21:17:08.365020 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:08.364970    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" event={"ID":"003dc2a1-a2b1-4659-99e8-49b2e73b50ca","Type":"ContainerStarted","Data":"ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd"}
Jul 01 21:17:09.373227 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:09.373184    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" event={"ID":"003dc2a1-a2b1-4659-99e8-49b2e73b50ca","Type":"ContainerStarted","Data":"798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e"}
Jul 01 21:17:09.374114 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:09.373508    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:11.387825 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:11.387774    2573 generic.go:356] "Generic (PLEG): container finished" podID="9979355a-45a2-4171-9164-8afa824810e7" containerID="477c85e8444bd0b64c90ae541d0ecd5626636165763504ea06ed6497a75264d8" exitCode=0
Jul 01 21:17:11.388242 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:11.387846    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" event={"ID":"9979355a-45a2-4171-9164-8afa824810e7","Type":"ContainerDied","Data":"477c85e8444bd0b64c90ae541d0ecd5626636165763504ea06ed6497a75264d8"}
Jul 01 21:17:12.395706 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:12.395606    2573 generic.go:356] "Generic (PLEG): container finished" podID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerID="798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e" exitCode=0
Jul 01 21:17:12.395706 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:12.395683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" event={"ID":"003dc2a1-a2b1-4659-99e8-49b2e73b50ca","Type":"ContainerDied","Data":"798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e"}
Jul 01 21:17:12.397846 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:12.397824    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" event={"ID":"9979355a-45a2-4171-9164-8afa824810e7","Type":"ContainerStarted","Data":"fc7a251634bdace80a7fb5b4c3f94f45b598fb93e279be128e7adf6d07f63396"}
Jul 01 21:17:13.406730 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:13.406691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" event={"ID":"003dc2a1-a2b1-4659-99e8-49b2e73b50ca","Type":"ContainerStarted","Data":"55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333"}
Jul 01 21:17:13.438272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:13.438185    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podStartSLOduration=3.3399823140000002 podStartE2EDuration="7.438163439s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.098181125s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:17:12.461500701 +0000 UTC m=+1865.372835338" watchObservedRunningTime="2026-07-01 21:17:13.438163439 +0000 UTC m=+1866.349498068"
Jul 01 21:17:13.438657 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:13.438619    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podStartSLOduration=3.023086866 podStartE2EDuration="7.438603789s" totalImagesPullingTime="779.679567ms" totalInitContainerRuntime="3.635837356s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:17:13.436509395 +0000 UTC m=+1866.347844022" watchObservedRunningTime="2026-07-01 21:17:13.438603789 +0000 UTC m=+1866.349938616"
Jul 01 21:17:16.768005 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:16.767969    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:16.768503 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:16.768056    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:16.769549 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:16.769507    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:17:16.813439 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:16.813404    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:16.813755 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:16.813716    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:17:16.814979 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:16.814949    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:17:17.454436 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:17.454406    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:17:24.466481 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:24.466444    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"]
Jul 01 21:17:26.768000 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:26.767952    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:17:26.813807 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:26.813764    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:17:36.768264 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:36.768158    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:17:36.813636 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:36.813590    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:17:39.494397 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:39.494349    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" containerID="cri-o://c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9" gracePeriod=45
Jul 01 21:17:40.250555 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.250525    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:17:40.358942 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.358917    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-dshm\") pod \"e83a902f-c8b7-4df3-a590-c4a5363287b9\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") "
Jul 01 21:17:40.359127 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.358952    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-tmp-dir\") pod \"e83a902f-c8b7-4df3-a590-c4a5363287b9\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") "
Jul 01 21:17:40.359127 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.358982    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83a902f-c8b7-4df3-a590-c4a5363287b9-tls-certs\") pod \"e83a902f-c8b7-4df3-a590-c4a5363287b9\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") "
Jul 01 21:17:40.359127 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.359008    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-home\") pod \"e83a902f-c8b7-4df3-a590-c4a5363287b9\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") "
Jul 01 21:17:40.359127 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.359102    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"e83a902f-c8b7-4df3-a590-c4a5363287b9\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") "
Jul 01 21:17:40.359349 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.359134    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-model-cache\") pod \"e83a902f-c8b7-4df3-a590-c4a5363287b9\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") "
Jul 01 21:17:40.359349 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.359203    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znrzb\" (UniqueName: \"kubernetes.io/projected/e83a902f-c8b7-4df3-a590-c4a5363287b9-kube-api-access-znrzb\") pod \"e83a902f-c8b7-4df3-a590-c4a5363287b9\" (UID: \"e83a902f-c8b7-4df3-a590-c4a5363287b9\") "
Jul 01 21:17:40.359643 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.359564    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e83a902f-c8b7-4df3-a590-c4a5363287b9" (UID: "e83a902f-c8b7-4df3-a590-c4a5363287b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:17:40.360237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.360208    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-home" (OuterVolumeSpecName: "home") pod "e83a902f-c8b7-4df3-a590-c4a5363287b9" (UID: "e83a902f-c8b7-4df3-a590-c4a5363287b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:17:40.361373 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.361351    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83a902f-c8b7-4df3-a590-c4a5363287b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e83a902f-c8b7-4df3-a590-c4a5363287b9" (UID: "e83a902f-c8b7-4df3-a590-c4a5363287b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:17:40.361614 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.361551    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-dshm" (OuterVolumeSpecName: "dshm") pod "e83a902f-c8b7-4df3-a590-c4a5363287b9" (UID: "e83a902f-c8b7-4df3-a590-c4a5363287b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:17:40.362111 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.362086    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83a902f-c8b7-4df3-a590-c4a5363287b9-kube-api-access-znrzb" (OuterVolumeSpecName: "kube-api-access-znrzb") pod "e83a902f-c8b7-4df3-a590-c4a5363287b9" (UID: "e83a902f-c8b7-4df3-a590-c4a5363287b9"). InnerVolumeSpecName "kube-api-access-znrzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:17:40.363902 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.363880    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f" (OuterVolumeSpecName: "kserve-pvc-source") pod "e83a902f-c8b7-4df3-a590-c4a5363287b9" (UID: "e83a902f-c8b7-4df3-a590-c4a5363287b9"). InnerVolumeSpecName "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 01 21:17:40.374696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.374670    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e83a902f-c8b7-4df3-a590-c4a5363287b9" (UID: "e83a902f-c8b7-4df3-a590-c4a5363287b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:17:40.460868 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.460820    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:17:40.461030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.460875    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:17:40.461030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.460896    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83a902f-c8b7-4df3-a590-c4a5363287b9-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:17:40.461030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.460906    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:17:40.461030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.460954    2573 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") on node \"ip-10-0-138-219.ec2.internal\" "
Jul 01 21:17:40.461030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.460968    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83a902f-c8b7-4df3-a590-c4a5363287b9-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:17:40.461030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.460981    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znrzb\" (UniqueName: \"kubernetes.io/projected/e83a902f-c8b7-4df3-a590-c4a5363287b9-kube-api-access-znrzb\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:17:40.491353 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.491330    2573 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f") on node "ip-10-0-138-219.ec2.internal"
Jul 01 21:17:40.561426 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.561397    2573 reconciler_common.go:299] "Volume detached for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\""
Jul 01 21:17:40.564610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.564583    2573 generic.go:356] "Generic (PLEG): container finished" podID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerID="c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9" exitCode=0
Jul 01 21:17:40.564727 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.564630    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"
Jul 01 21:17:40.564727 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.564671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" event={"ID":"e83a902f-c8b7-4df3-a590-c4a5363287b9","Type":"ContainerDied","Data":"c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9"}
Jul 01 21:17:40.564849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.564739    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" event={"ID":"e83a902f-c8b7-4df3-a590-c4a5363287b9","Type":"ContainerDied","Data":"936cdc5c5f291f2f67bbcf3eae4752ff4618b718bd782c9173e426226c0d0620"}
Jul 01 21:17:40.564849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.564773    2573 scope.go:117] "RemoveContainer" containerID="c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9"
Jul 01 21:17:40.575446 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.575431    2573 scope.go:117] "RemoveContainer" containerID="c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9"
Jul 01 21:17:40.575705 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:17:40.575688    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9\": container with ID starting with c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9 not found: ID does not exist" containerID="c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9"
Jul 01 21:17:40.575773 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.575711    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9"} err="failed to get container status \"c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9\": rpc error: code = NotFound desc = could not find container \"c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9\": container with ID starting with c7a2df80307a4559cd91fe2ed80732fb157afcfb1fa574893f3d4f1e2376acd9 not found: ID does not exist"
Jul 01 21:17:40.592943 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.592916    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"]
Jul 01 21:17:40.599122 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:40.599103    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5"]
Jul 01 21:17:41.210992 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:41.210946    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 01 21:17:41.594353 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:41.594309    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" path="/var/lib/kubelet/pods/e83a902f-c8b7-4df3-a590-c4a5363287b9/volumes"
Jul 01 21:17:46.768836 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:46.768788    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:17:46.813211 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:46.813176    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:17:56.768969 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:56.768904    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:17:56.813189 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:56.813139    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:17:59.535610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.535576    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"]
Jul 01 21:17:59.536242 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.535931    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main"
Jul 01 21:17:59.536242 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.535944    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main"
Jul 01 21:17:59.536242 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.536011    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83a902f-c8b7-4df3-a590-c4a5363287b9" containerName="main"
Jul 01 21:17:59.539239 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.539211    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.543577 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.543548    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-bc94h\""
Jul 01 21:17:59.543938 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.543918    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 01 21:17:59.557697 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.557669    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"]
Jul 01 21:17:59.564504 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.564478    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"]
Jul 01 21:17:59.567263 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.567244    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.578556 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.578530    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"]
Jul 01 21:17:59.637396 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.637358    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.637564 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.637422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.637564 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.637445    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9506d770-6842-45fb-8489-6f35a5b46751-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.637564 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.637484    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdv4c\" (UniqueName: \"kubernetes.io/projected/9506d770-6842-45fb-8489-6f35a5b46751-kube-api-access-qdv4c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.637564 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.637538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.637782 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.637580    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.738459 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.738624 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738465    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee58395-0ba2-4058-83e4-892686bf16e5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.738624 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.738624 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.738624 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738591    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9506d770-6842-45fb-8489-6f35a5b46751-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.738876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdv4c\" (UniqueName: \"kubernetes.io/projected/9506d770-6842-45fb-8489-6f35a5b46751-kube-api-access-qdv4c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.738876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.738876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738685    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.738876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2q7z\" (UniqueName: \"kubernetes.io/projected/4ee58395-0ba2-4058-83e4-892686bf16e5-kube-api-access-v2q7z\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.738876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.738876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.738876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.738876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738847    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.739274 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738938    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.739274 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.738982    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.741071 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.741049    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.741317 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.741297    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9506d770-6842-45fb-8489-6f35a5b46751-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.750306 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.750286    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdv4c\" (UniqueName: \"kubernetes.io/projected/9506d770-6842-45fb-8489-6f35a5b46751-kube-api-access-qdv4c\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:17:59.839713 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.839681    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.839898 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.839769    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee58395-0ba2-4058-83e4-892686bf16e5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.839979 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.839905    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.839979 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.839934    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.839979 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.839971    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2q7z\" (UniqueName: \"kubernetes.io/projected/4ee58395-0ba2-4058-83e4-892686bf16e5-kube-api-access-v2q7z\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.840138 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.840018    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.840390 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.840336    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.840390 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.840360    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.840569 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.840396    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.842072 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.842051    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.842460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.842442    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee58395-0ba2-4058-83e4-892686bf16e5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:17:59.853790 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:17:59.853754    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2q7z\" (UniqueName: \"kubernetes.io/projected/4ee58395-0ba2-4058-83e4-892686bf16e5-kube-api-access-v2q7z\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:18:06.768217 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:06.768166    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:18:06.813197 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:06.813163    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:18:07.109731 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.109692    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:07.116424 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.116398    2573 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:07.210573 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.210535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:07.210743 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.210657    2573 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:07.214447 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.214425    2573 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:07.306755 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.306726    2573 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a7984d75592a60f75acc48715216990326d59cb841472d1b5739e644ba9a7d8d/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:07.334102 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.334071    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:07.351890 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.351862    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:07.412566 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.412524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:18:07.412772 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.412751    2573 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:18:07.416967 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.416886    2573 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:18:07.434289 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.434254    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:18:07.517783 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.517758    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"]
Jul 01 21:18:07.519782 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:18:07.519758    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9506d770_6842_45fb_8489_6f35a5b46751.slice/crio-e580ae700c12aaf990ccccf6f683e749bdfbba283029a59bc1b75465e1fbf92b WatchSource:0}: Error finding container e580ae700c12aaf990ccccf6f683e749bdfbba283029a59bc1b75465e1fbf92b: Status 404 returned error can't find the container with id e580ae700c12aaf990ccccf6f683e749bdfbba283029a59bc1b75465e1fbf92b
Jul 01 21:18:07.679800 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.679706    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:18:07.703571 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.703002    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" event={"ID":"9506d770-6842-45fb-8489-6f35a5b46751","Type":"ContainerStarted","Data":"fb96637aff0ddf465e1f00866e113cd8f0b26c7674e27b79314793f4c872c055"}
Jul 01 21:18:07.703733 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.703641    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:07.703733 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.703672    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" event={"ID":"9506d770-6842-45fb-8489-6f35a5b46751","Type":"ContainerStarted","Data":"e580ae700c12aaf990ccccf6f683e749bdfbba283029a59bc1b75465e1fbf92b"}
Jul 01 21:18:07.848474 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:07.848438    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"]
Jul 01 21:18:07.854547 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:18:07.854517    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee58395_0ba2_4058_83e4_892686bf16e5.slice/crio-1de5de1723096a8691b41c9fee3cfbcd8526743414ad4136b4d22908a244fdeb WatchSource:0}: Error finding container 1de5de1723096a8691b41c9fee3cfbcd8526743414ad4136b4d22908a244fdeb: Status 404 returned error can't find the container with id 1de5de1723096a8691b41c9fee3cfbcd8526743414ad4136b4d22908a244fdeb
Jul 01 21:18:08.711912 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:08.711848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" event={"ID":"9506d770-6842-45fb-8489-6f35a5b46751","Type":"ContainerStarted","Data":"954e747baac21499c4813deb9238bc8e58fc0a43ea7841e967ae84520460a03b"}
Jul 01 21:18:08.713630 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:08.713603    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" event={"ID":"4ee58395-0ba2-4058-83e4-892686bf16e5","Type":"ContainerStarted","Data":"fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f"}
Jul 01 21:18:08.713753 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:08.713651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" event={"ID":"4ee58395-0ba2-4058-83e4-892686bf16e5","Type":"ContainerStarted","Data":"1de5de1723096a8691b41c9fee3cfbcd8526743414ad4136b4d22908a244fdeb"}
Jul 01 21:18:08.740369 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:08.740311    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podStartSLOduration=9.740289465 podStartE2EDuration="9.740289465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:18:08.737433304 +0000 UTC m=+1921.648767958" watchObservedRunningTime="2026-07-01 21:18:08.740289465 +0000 UTC m=+1921.651624093"
Jul 01 21:18:08.773532 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:08.773429    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podStartSLOduration=9.773408467 podStartE2EDuration="9.773408467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:18:08.772464269 +0000 UTC m=+1921.683798896" watchObservedRunningTime="2026-07-01 21:18:08.773408467 +0000 UTC m=+1921.684743094"
Jul 01 21:18:16.768200 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:16.768144    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:18:16.813614 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:16.813571    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:18:17.352491 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:17.352452    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:17.352669 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:17.352523    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:17.354595 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:17.354557    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:18:17.379257 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:17.379226    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:18:17.680193 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:17.680087    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:18:17.680372 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:17.680276    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:18:17.681816 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:17.681774    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:18:26.769035 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:26.768986    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:18:26.813111 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:26.813058    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:18:27.353294 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:27.353247    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:18:27.680227 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:27.680120    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:18:36.768241 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:36.768192    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:18:36.813422 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:36.813381    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:18:37.353294 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:37.353239    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:18:37.681358 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:37.681243    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:18:46.768944 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:46.768870    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:18:46.813488 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:46.813432    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:18:47.353245 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:47.353199    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:18:47.680180 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:47.680087    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:18:56.768520 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:56.768473    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:18:56.812891 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:56.812833    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:18:57.352697 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:57.352649    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:18:57.681062 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:18:57.680962    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:19:06.768993 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:06.768876    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:19:06.813729 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:06.813684    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:19:07.352994 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:07.352934    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:19:07.681012 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:07.680896    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:19:16.768027 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:16.767972    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:19:16.813185 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:16.813143    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:19:17.353370 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:17.353301    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:19:17.680570 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:17.680460    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:19:26.768280 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:26.768234    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:19:26.812767 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:26.812727    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:19:27.352535 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:27.352488    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:19:27.680672 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:27.680573    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:19:36.768711 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:36.768663    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:19:36.813295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:36.813253    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:19:37.352808 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:37.352752    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:19:37.680618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:37.680528    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:19:46.768260 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:46.768207    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:19:46.813547 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:46.813505    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:19:47.353076 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:47.353032    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:19:47.680435 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:47.680348    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:19:56.768954 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:56.768902    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:19:56.813004 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:56.812971    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 01 21:19:57.353178 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:57.353138    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:19:57.680300 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:19:57.680204    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:20:06.768801 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:06.768735    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Jul 01 21:20:06.822913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:06.822881    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:20:06.830570 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:06.830537    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:20:07.352503 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:07.352449    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:20:07.680275 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:07.680189    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:20:16.777946 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:16.777914    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:20:16.794770 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:16.794745    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:20:17.352698 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:17.352654    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:20:17.680544 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:17.680436    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:20:27.353260 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:27.353212    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:20:27.680160 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:27.680063    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:20:37.353311 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:37.353220    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:20:37.681295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:37.681198    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:20:39.257939 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:39.257906    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"]
Jul 01 21:20:39.261500 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:39.261472    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"]
Jul 01 21:20:47.352415 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:47.352368    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:20:47.680363 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:47.680272    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:20:54.282152 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:54.282080    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main" containerID="cri-o://55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333" gracePeriod=45
Jul 01 21:20:54.284115 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:54.284077    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main" containerID="cri-o://fc7a251634bdace80a7fb5b4c3f94f45b598fb93e279be128e7adf6d07f63396" gracePeriod=45
Jul 01 21:20:55.596623 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.596306    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="llm-d-routing-sidecar" containerID="cri-o://ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd" gracePeriod=44
Jul 01 21:20:55.640149 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.640111    2573 generic.go:356] "Generic (PLEG): container finished" podID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerID="55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333" exitCode=0
Jul 01 21:20:55.640322 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.640166    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" event={"ID":"003dc2a1-a2b1-4659-99e8-49b2e73b50ca","Type":"ContainerDied","Data":"55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333"}
Jul 01 21:20:55.642375 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.642354    2573 generic.go:356] "Generic (PLEG): container finished" podID="9979355a-45a2-4171-9164-8afa824810e7" containerID="fc7a251634bdace80a7fb5b4c3f94f45b598fb93e279be128e7adf6d07f63396" exitCode=0
Jul 01 21:20:55.642487 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.642424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" event={"ID":"9979355a-45a2-4171-9164-8afa824810e7","Type":"ContainerDied","Data":"fc7a251634bdace80a7fb5b4c3f94f45b598fb93e279be128e7adf6d07f63396"}
Jul 01 21:20:55.742713 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.742651    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:20:55.828724 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.828694    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-home\") pod \"9979355a-45a2-4171-9164-8afa824810e7\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") "
Jul 01 21:20:55.828904 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.828736    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-tmp-dir\") pod \"9979355a-45a2-4171-9164-8afa824810e7\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") "
Jul 01 21:20:55.828904 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.828792    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-kserve-provision-location\") pod \"9979355a-45a2-4171-9164-8afa824810e7\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") "
Jul 01 21:20:55.828904 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.828837    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-dshm\") pod \"9979355a-45a2-4171-9164-8afa824810e7\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") "
Jul 01 21:20:55.828904 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.828877    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9979355a-45a2-4171-9164-8afa824810e7-tls-certs\") pod \"9979355a-45a2-4171-9164-8afa824810e7\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") "
Jul 01 21:20:55.828904 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.828902    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxnb9\" (UniqueName: \"kubernetes.io/projected/9979355a-45a2-4171-9164-8afa824810e7-kube-api-access-xxnb9\") pod \"9979355a-45a2-4171-9164-8afa824810e7\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") "
Jul 01 21:20:55.829206 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.828990    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-model-cache\") pod \"9979355a-45a2-4171-9164-8afa824810e7\" (UID: \"9979355a-45a2-4171-9164-8afa824810e7\") "
Jul 01 21:20:55.829330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.829307    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-home" (OuterVolumeSpecName: "home") pod "9979355a-45a2-4171-9164-8afa824810e7" (UID: "9979355a-45a2-4171-9164-8afa824810e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:55.829734 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.829692    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "9979355a-45a2-4171-9164-8afa824810e7" (UID: "9979355a-45a2-4171-9164-8afa824810e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:55.832112 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.832074    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9979355a-45a2-4171-9164-8afa824810e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9979355a-45a2-4171-9164-8afa824810e7" (UID: "9979355a-45a2-4171-9164-8afa824810e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:20:55.832222 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.832122    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-dshm" (OuterVolumeSpecName: "dshm") pod "9979355a-45a2-4171-9164-8afa824810e7" (UID: "9979355a-45a2-4171-9164-8afa824810e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:55.832281 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.832211    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9979355a-45a2-4171-9164-8afa824810e7-kube-api-access-xxnb9" (OuterVolumeSpecName: "kube-api-access-xxnb9") pod "9979355a-45a2-4171-9164-8afa824810e7" (UID: "9979355a-45a2-4171-9164-8afa824810e7"). InnerVolumeSpecName "kube-api-access-xxnb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:20:55.848371 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.848340    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9979355a-45a2-4171-9164-8afa824810e7" (UID: "9979355a-45a2-4171-9164-8afa824810e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:55.862975 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.862949    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:20:55.908628 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.908591    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9979355a-45a2-4171-9164-8afa824810e7" (UID: "9979355a-45a2-4171-9164-8afa824810e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:55.930111 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930084    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-dshm\") pod \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") "
Jul 01 21:20:55.930248 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930134    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tls-certs\") pod \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") "
Jul 01 21:20:55.930248 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930171    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-home\") pod \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") "
Jul 01 21:20:55.930248 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930191    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvls8\" (UniqueName: \"kubernetes.io/projected/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kube-api-access-pvls8\") pod \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") "
Jul 01 21:20:55.930248 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930223    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kserve-provision-location\") pod \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") "
Jul 01 21:20:55.930448 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930258    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tmp-dir\") pod \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") "
Jul 01 21:20:55.930448 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930309    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-model-cache\") pod \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\" (UID: \"003dc2a1-a2b1-4659-99e8-49b2e73b50ca\") "
Jul 01 21:20:55.930613 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930591    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:55.930796 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930781    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:55.930950 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930930    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9979355a-45a2-4171-9164-8afa824810e7-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:55.931077 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930954    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxnb9\" (UniqueName: \"kubernetes.io/projected/9979355a-45a2-4171-9164-8afa824810e7-kube-api-access-xxnb9\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:55.931077 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930968    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:55.931077 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930983    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:55.931077 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930995    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9979355a-45a2-4171-9164-8afa824810e7-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:55.931077 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.930609    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "003dc2a1-a2b1-4659-99e8-49b2e73b50ca" (UID: "003dc2a1-a2b1-4659-99e8-49b2e73b50ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:55.931521 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.931497    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-home" (OuterVolumeSpecName: "home") pod "003dc2a1-a2b1-4659-99e8-49b2e73b50ca" (UID: "003dc2a1-a2b1-4659-99e8-49b2e73b50ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:55.932817 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.932776    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-dshm" (OuterVolumeSpecName: "dshm") pod "003dc2a1-a2b1-4659-99e8-49b2e73b50ca" (UID: "003dc2a1-a2b1-4659-99e8-49b2e73b50ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:55.932937 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.932876    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "003dc2a1-a2b1-4659-99e8-49b2e73b50ca" (UID: "003dc2a1-a2b1-4659-99e8-49b2e73b50ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:20:55.932937 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.932908    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kube-api-access-pvls8" (OuterVolumeSpecName: "kube-api-access-pvls8") pod "003dc2a1-a2b1-4659-99e8-49b2e73b50ca" (UID: "003dc2a1-a2b1-4659-99e8-49b2e73b50ca"). InnerVolumeSpecName "kube-api-access-pvls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:20:55.942926 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.942901    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "003dc2a1-a2b1-4659-99e8-49b2e73b50ca" (UID: "003dc2a1-a2b1-4659-99e8-49b2e73b50ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:55.990422 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:55.990356    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "003dc2a1-a2b1-4659-99e8-49b2e73b50ca" (UID: "003dc2a1-a2b1-4659-99e8-49b2e73b50ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:20:56.031770 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.031679    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:56.031770 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.031707    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:56.031770 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.031716    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:56.031770 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.031724    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:56.031770 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.031732    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:56.031770 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.031741    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvls8\" (UniqueName: \"kubernetes.io/projected/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kube-api-access-pvls8\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:56.031770 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.031748    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/003dc2a1-a2b1-4659-99e8-49b2e73b50ca-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:20:56.649217 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.649186    2573 generic.go:356] "Generic (PLEG): container finished" podID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerID="ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd" exitCode=0
Jul 01 21:20:56.649694 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.649242    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"
Jul 01 21:20:56.649694 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.649272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" event={"ID":"003dc2a1-a2b1-4659-99e8-49b2e73b50ca","Type":"ContainerDied","Data":"ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd"}
Jul 01 21:20:56.649694 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.649345    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h" event={"ID":"003dc2a1-a2b1-4659-99e8-49b2e73b50ca","Type":"ContainerDied","Data":"01ff1cd4d342bfac59b7c4ce82ae0a61a9f71973bcfdcbb41f16a7e7fb075325"}
Jul 01 21:20:56.649694 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.649381    2573 scope.go:117] "RemoveContainer" containerID="55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333"
Jul 01 21:20:56.650978 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.650958    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"
Jul 01 21:20:56.651101 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.650991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx" event={"ID":"9979355a-45a2-4171-9164-8afa824810e7","Type":"ContainerDied","Data":"a5c718d987e683082dba3c2986b97412a98bc38445af4b3e6f590ad94b73a0f9"}
Jul 01 21:20:56.667685 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.667511    2573 scope.go:117] "RemoveContainer" containerID="798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e"
Jul 01 21:20:56.678052 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.678036    2573 scope.go:117] "RemoveContainer" containerID="ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd"
Jul 01 21:20:56.685267 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.685247    2573 scope.go:117] "RemoveContainer" containerID="55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333"
Jul 01 21:20:56.685556 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:20:56.685538    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333\": container with ID starting with 55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333 not found: ID does not exist" containerID="55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333"
Jul 01 21:20:56.685634 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.685568    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333"} err="failed to get container status \"55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333\": rpc error: code = NotFound desc = could not find container \"55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333\": container with ID starting with 55dce9ad17836fec620916f262d1db9d631080cd870a35e313d778ca9a118333 not found: ID does not exist"
Jul 01 21:20:56.685634 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.685593    2573 scope.go:117] "RemoveContainer" containerID="798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e"
Jul 01 21:20:56.686066 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:20:56.686022    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e\": container with ID starting with 798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e not found: ID does not exist" containerID="798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e"
Jul 01 21:20:56.686160 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.686064    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e"} err="failed to get container status \"798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e\": rpc error: code = NotFound desc = could not find container \"798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e\": container with ID starting with 798be3c908b7eb671fd71fa30d7e6479150f005c8cc300cb66e04e3547bd1b5e not found: ID does not exist"
Jul 01 21:20:56.686160 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.686085    2573 scope.go:117] "RemoveContainer" containerID="ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd"
Jul 01 21:20:56.686400 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:20:56.686371    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd\": container with ID starting with ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd not found: ID does not exist" containerID="ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd"
Jul 01 21:20:56.686460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.686404    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd"} err="failed to get container status \"ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd\": rpc error: code = NotFound desc = could not find container \"ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd\": container with ID starting with ae8da59e49f4a26d83c165d54fb859dbf73bd20942be50710804f783c9dd14cd not found: ID does not exist"
Jul 01 21:20:56.686460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.686417    2573 scope.go:117] "RemoveContainer" containerID="fc7a251634bdace80a7fb5b4c3f94f45b598fb93e279be128e7adf6d07f63396"
Jul 01 21:20:56.688295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.688275    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"]
Jul 01 21:20:56.694321 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.694297    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx"]
Jul 01 21:20:56.694660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.694648    2573 scope.go:117] "RemoveContainer" containerID="477c85e8444bd0b64c90ae541d0ecd5626636165763504ea06ed6497a75264d8"
Jul 01 21:20:56.722568 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.722548    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"]
Jul 01 21:20:56.724764 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:56.724746    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h"]
Jul 01 21:20:57.352758 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:57.352717    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:20:57.591894 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:57.591836    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" path="/var/lib/kubelet/pods/003dc2a1-a2b1-4659-99e8-49b2e73b50ca/volumes"
Jul 01 21:20:57.592585 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:57.592564    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9979355a-45a2-4171-9164-8afa824810e7" path="/var/lib/kubelet/pods/9979355a-45a2-4171-9164-8afa824810e7/volumes"
Jul 01 21:20:57.680313 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:20:57.680216    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:21:06.136339 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136307    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"]
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136636    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="storage-initializer"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136648    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="storage-initializer"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136663    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136670    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136681    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="storage-initializer"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136687    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="storage-initializer"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136695    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136700    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136712    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="llm-d-routing-sidecar"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136717    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="llm-d-routing-sidecar"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136775    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="main"
Jul 01 21:21:06.136778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136784    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="003dc2a1-a2b1-4659-99e8-49b2e73b50ca" containerName="llm-d-routing-sidecar"
Jul 01 21:21:06.137182 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.136790    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9979355a-45a2-4171-9164-8afa824810e7" containerName="main"
Jul 01 21:21:06.141327 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.141305    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.146712 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.146686    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:21:06.147397 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.147376    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-k27mh\""
Jul 01 21:21:06.148597 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.148576    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 01 21:21:06.155200 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.155175    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"]
Jul 01 21:21:06.160673 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.160654    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"]
Jul 01 21:21:06.164172 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.164157    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.183337 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.183310    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"]
Jul 01 21:21:06.229131 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.229315 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.229315 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229179    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.229315 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.229315 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229254    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.229315 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229306    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5qv\" (UniqueName: \"kubernetes.io/projected/ab193660-208e-4e20-a524-a7abb9a93f6c-kube-api-access-qp5qv\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.229617 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229344    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.229617 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqmg\" (UniqueName: \"kubernetes.io/projected/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kube-api-access-hhqmg\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.229617 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229404    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-home\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.229617 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229500    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.229617 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229561    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab193660-208e-4e20-a524-a7abb9a93f6c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.229617 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229606    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-dshm\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.229996 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229632    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.229996 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.229665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.330238 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.330388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5qv\" (UniqueName: \"kubernetes.io/projected/ab193660-208e-4e20-a524-a7abb9a93f6c-kube-api-access-qp5qv\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.330388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.330388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqmg\" (UniqueName: \"kubernetes.io/projected/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kube-api-access-hhqmg\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.330388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-home\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.330388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.330688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab193660-208e-4e20-a524-a7abb9a93f6c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.330739 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330710    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-home\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.330787 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330712    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.330787 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330746    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-dshm\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.330787 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330668    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.330787 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330751    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.330787 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.331071 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.331071 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.331071 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.330912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.331227 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.331102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.331227 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.331133    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.331227 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.331137    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.331227 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.331193    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.331442 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.331259    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.331505 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.331465    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.333205 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.333178    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.333328 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.333224    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab193660-208e-4e20-a524-a7abb9a93f6c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.333460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.333437    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-dshm\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.333876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.333837    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.343422 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.343394    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5qv\" (UniqueName: \"kubernetes.io/projected/ab193660-208e-4e20-a524-a7abb9a93f6c-kube-api-access-qp5qv\") pod \"custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.343585 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.343558    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqmg\" (UniqueName: \"kubernetes.io/projected/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kube-api-access-hhqmg\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.451084 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.451006    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:06.474575 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.474549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:06.651878 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.651833    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"]
Jul 01 21:21:06.654678 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:21:06.654642    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab193660_208e_4e20_a524_a7abb9a93f6c.slice/crio-5a753a624105b8644389640923a1e5abf38d446d7e196634624d7ef4a78d1f97 WatchSource:0}: Error finding container 5a753a624105b8644389640923a1e5abf38d446d7e196634624d7ef4a78d1f97: Status 404 returned error can't find the container with id 5a753a624105b8644389640923a1e5abf38d446d7e196634624d7ef4a78d1f97
Jul 01 21:21:06.685719 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.685699    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"]
Jul 01 21:21:06.693319 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:21:06.693277    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bfb1e1_e817_47ab_ae25_c5a71b828396.slice/crio-631dcccfd95506a92af8d1d01b5ac3d4fe37be1e6af3284ea2e2754c6221dbd0 WatchSource:0}: Error finding container 631dcccfd95506a92af8d1d01b5ac3d4fe37be1e6af3284ea2e2754c6221dbd0: Status 404 returned error can't find the container with id 631dcccfd95506a92af8d1d01b5ac3d4fe37be1e6af3284ea2e2754c6221dbd0
Jul 01 21:21:06.707892 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.707849    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" event={"ID":"ab193660-208e-4e20-a524-a7abb9a93f6c","Type":"ContainerStarted","Data":"5a753a624105b8644389640923a1e5abf38d446d7e196634624d7ef4a78d1f97"}
Jul 01 21:21:06.708900 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:06.708879    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" event={"ID":"c0bfb1e1-e817-47ab-ae25-c5a71b828396","Type":"ContainerStarted","Data":"631dcccfd95506a92af8d1d01b5ac3d4fe37be1e6af3284ea2e2754c6221dbd0"}
Jul 01 21:21:07.352590 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:07.352540    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Jul 01 21:21:07.680192 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:07.680103    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Jul 01 21:21:07.689942 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:07.689921    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:21:07.691513 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:07.691488    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:21:07.708413 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:07.708388    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:21:07.710847 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:07.710828    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:21:07.716196 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:07.716167    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" event={"ID":"ab193660-208e-4e20-a524-a7abb9a93f6c","Type":"ContainerStarted","Data":"5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521"}
Jul 01 21:21:07.716304 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:07.716274    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:07.717953 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:07.717931    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" event={"ID":"c0bfb1e1-e817-47ab-ae25-c5a71b828396","Type":"ContainerStarted","Data":"ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf"}
Jul 01 21:21:08.725769 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:08.725733    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" event={"ID":"ab193660-208e-4e20-a524-a7abb9a93f6c","Type":"ContainerStarted","Data":"5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a"}
Jul 01 21:21:11.742921 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:11.742880    2573 generic.go:356] "Generic (PLEG): container finished" podID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerID="ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf" exitCode=0
Jul 01 21:21:11.743282 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:11.742887    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" event={"ID":"c0bfb1e1-e817-47ab-ae25-c5a71b828396","Type":"ContainerDied","Data":"ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf"}
Jul 01 21:21:12.750897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:12.750836    2573 generic.go:356] "Generic (PLEG): container finished" podID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerID="5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a" exitCode=0
Jul 01 21:21:12.750897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:12.750892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" event={"ID":"ab193660-208e-4e20-a524-a7abb9a93f6c","Type":"ContainerDied","Data":"5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a"}
Jul 01 21:21:12.753481 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:12.753456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" event={"ID":"c0bfb1e1-e817-47ab-ae25-c5a71b828396","Type":"ContainerStarted","Data":"7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157"}
Jul 01 21:21:12.817290 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:12.817248    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podStartSLOduration=2.454879653 podStartE2EDuration="6.817232616s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.362352963s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:21:12.814370676 +0000 UTC m=+2105.725705284" watchObservedRunningTime="2026-07-01 21:21:12.817232616 +0000 UTC m=+2105.728567243"
Jul 01 21:21:13.760540 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:13.760497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" event={"ID":"ab193660-208e-4e20-a524-a7abb9a93f6c","Type":"ContainerStarted","Data":"ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf"}
Jul 01 21:21:13.795007 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:13.794925    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podStartSLOduration=3.781223868 podStartE2EDuration="7.794905s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.013681132s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:21:13.793414905 +0000 UTC m=+2106.704749546" watchObservedRunningTime="2026-07-01 21:21:13.794905 +0000 UTC m=+2106.706239627"
Jul 01 21:21:16.451233 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:16.451192    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:16.451685 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:16.451264    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:16.452925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:16.452890    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:21:16.464460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:16.464438    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:21:16.474827 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:16.474806    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:16.474964 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:16.474871    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:21:16.476240 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:16.476210    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:21:17.368519 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:17.368485    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:21:17.388442 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:17.388407    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:21:17.690263 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:17.690171    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:21:17.702297 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:17.702268    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:21:26.451902 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:26.451841    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:21:26.476010 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:26.475970    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:21:27.859196 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:27.859163    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"]
Jul 01 21:21:27.870235 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:27.870203    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"]
Jul 01 21:21:36.452124 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:36.452055    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:21:36.475104 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:36.475066    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:21:42.896917 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:42.896803    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main" containerID="cri-o://954e747baac21499c4813deb9238bc8e58fc0a43ea7841e967ae84520460a03b" gracePeriod=45
Jul 01 21:21:42.905500 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:42.905404    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main" containerID="cri-o://fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f" gracePeriod=45
Jul 01 21:21:43.723148 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.723112    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="llm-d-routing-sidecar" containerID="cri-o://fb96637aff0ddf465e1f00866e113cd8f0b26c7674e27b79314793f4c872c055" gracePeriod=45
Jul 01 21:21:43.884704 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.884670    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:21:43.925837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.924905    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-model-cache\") pod \"4ee58395-0ba2-4058-83e4-892686bf16e5\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") "
Jul 01 21:21:43.925837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.924962    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2q7z\" (UniqueName: \"kubernetes.io/projected/4ee58395-0ba2-4058-83e4-892686bf16e5-kube-api-access-v2q7z\") pod \"4ee58395-0ba2-4058-83e4-892686bf16e5\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") "
Jul 01 21:21:43.925837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.925024    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-tmp-dir\") pod \"4ee58395-0ba2-4058-83e4-892686bf16e5\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") "
Jul 01 21:21:43.925837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.925052    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-home\") pod \"4ee58395-0ba2-4058-83e4-892686bf16e5\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") "
Jul 01 21:21:43.925837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.925087    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee58395-0ba2-4058-83e4-892686bf16e5-tls-certs\") pod \"4ee58395-0ba2-4058-83e4-892686bf16e5\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") "
Jul 01 21:21:43.925837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.925123    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-dshm\") pod \"4ee58395-0ba2-4058-83e4-892686bf16e5\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") "
Jul 01 21:21:43.925837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.925222    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"4ee58395-0ba2-4058-83e4-892686bf16e5\" (UID: \"4ee58395-0ba2-4058-83e4-892686bf16e5\") "
Jul 01 21:21:43.926870 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.926082    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ee58395-0ba2-4058-83e4-892686bf16e5" (UID: "4ee58395-0ba2-4058-83e4-892686bf16e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:21:43.926870 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.926762    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-home" (OuterVolumeSpecName: "home") pod "4ee58395-0ba2-4058-83e4-892686bf16e5" (UID: "4ee58395-0ba2-4058-83e4-892686bf16e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:21:43.928783 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.928756    2573 generic.go:356] "Generic (PLEG): container finished" podID="9506d770-6842-45fb-8489-6f35a5b46751" containerID="954e747baac21499c4813deb9238bc8e58fc0a43ea7841e967ae84520460a03b" exitCode=0
Jul 01 21:21:43.928783 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.928784    2573 generic.go:356] "Generic (PLEG): container finished" podID="9506d770-6842-45fb-8489-6f35a5b46751" containerID="fb96637aff0ddf465e1f00866e113cd8f0b26c7674e27b79314793f4c872c055" exitCode=0
Jul 01 21:21:43.929030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.928996    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" event={"ID":"9506d770-6842-45fb-8489-6f35a5b46751","Type":"ContainerDied","Data":"954e747baac21499c4813deb9238bc8e58fc0a43ea7841e967ae84520460a03b"}
Jul 01 21:21:43.929155 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.929055    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" event={"ID":"9506d770-6842-45fb-8489-6f35a5b46751","Type":"ContainerDied","Data":"fb96637aff0ddf465e1f00866e113cd8f0b26c7674e27b79314793f4c872c055"}
Jul 01 21:21:43.929368 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.929327    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee58395-0ba2-4058-83e4-892686bf16e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ee58395-0ba2-4058-83e4-892686bf16e5" (UID: "4ee58395-0ba2-4058-83e4-892686bf16e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:21:43.929561 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.929535    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee58395-0ba2-4058-83e4-892686bf16e5-kube-api-access-v2q7z" (OuterVolumeSpecName: "kube-api-access-v2q7z") pod "4ee58395-0ba2-4058-83e4-892686bf16e5" (UID: "4ee58395-0ba2-4058-83e4-892686bf16e5"). InnerVolumeSpecName "kube-api-access-v2q7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:21:43.930281 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.930258    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f" (OuterVolumeSpecName: "kserve-pvc-source") pod "4ee58395-0ba2-4058-83e4-892686bf16e5" (UID: "4ee58395-0ba2-4058-83e4-892686bf16e5"). InnerVolumeSpecName "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 01 21:21:43.930770 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.930747    2573 generic.go:356] "Generic (PLEG): container finished" podID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerID="fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f" exitCode=0
Jul 01 21:21:43.930848 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.930792    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" event={"ID":"4ee58395-0ba2-4058-83e4-892686bf16e5","Type":"ContainerDied","Data":"fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f"}
Jul 01 21:21:43.930848 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.930832    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp" event={"ID":"4ee58395-0ba2-4058-83e4-892686bf16e5","Type":"ContainerDied","Data":"1de5de1723096a8691b41c9fee3cfbcd8526743414ad4136b4d22908a244fdeb"}
Jul 01 21:21:43.930961 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.930876    2573 scope.go:117] "RemoveContainer" containerID="fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f"
Jul 01 21:21:43.931349 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.931024    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"
Jul 01 21:21:43.931448 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.931367    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-dshm" (OuterVolumeSpecName: "dshm") pod "4ee58395-0ba2-4058-83e4-892686bf16e5" (UID: "4ee58395-0ba2-4058-83e4-892686bf16e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:21:43.940472 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.940455    2573 scope.go:117] "RemoveContainer" containerID="fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f"
Jul 01 21:21:43.940758 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:21:43.940725    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f\": container with ID starting with fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f not found: ID does not exist" containerID="fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f"
Jul 01 21:21:43.940844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.940762    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f"} err="failed to get container status \"fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f\": rpc error: code = NotFound desc = could not find container \"fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f\": container with ID starting with fc6d5e7bfc3b5d2a23a3868a5e2bf94355cadf2d969b241adb7420196c21b61f not found: ID does not exist"
Jul 01 21:21:43.943149 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.943124    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ee58395-0ba2-4058-83e4-892686bf16e5" (UID: "4ee58395-0ba2-4058-83e4-892686bf16e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:21:43.985181 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:43.985158    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:21:44.026012 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.025988    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9506d770-6842-45fb-8489-6f35a5b46751-tls-certs\") pod \"9506d770-6842-45fb-8489-6f35a5b46751\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") "
Jul 01 21:21:44.026168 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026029    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-home\") pod \"9506d770-6842-45fb-8489-6f35a5b46751\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") "
Jul 01 21:21:44.026168 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026068    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-model-cache\") pod \"9506d770-6842-45fb-8489-6f35a5b46751\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") "
Jul 01 21:21:44.026168 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026089    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-tmp-dir\") pod \"9506d770-6842-45fb-8489-6f35a5b46751\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") "
Jul 01 21:21:44.026348 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026211    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"9506d770-6842-45fb-8489-6f35a5b46751\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") "
Jul 01 21:21:44.026348 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026248    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdv4c\" (UniqueName: \"kubernetes.io/projected/9506d770-6842-45fb-8489-6f35a5b46751-kube-api-access-qdv4c\") pod \"9506d770-6842-45fb-8489-6f35a5b46751\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") "
Jul 01 21:21:44.026348 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026272    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-dshm\") pod \"9506d770-6842-45fb-8489-6f35a5b46751\" (UID: \"9506d770-6842-45fb-8489-6f35a5b46751\") "
Jul 01 21:21:44.026552 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026532    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.026618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026557    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.026618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026570    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee58395-0ba2-4058-83e4-892686bf16e5-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.026618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026583    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.026618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026594    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee58395-0ba2-4058-83e4-892686bf16e5-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.026618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026606    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2q7z\" (UniqueName: \"kubernetes.io/projected/4ee58395-0ba2-4058-83e4-892686bf16e5-kube-api-access-v2q7z\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.026889 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.026777    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-home" (OuterVolumeSpecName: "home") pod "9506d770-6842-45fb-8489-6f35a5b46751" (UID: "9506d770-6842-45fb-8489-6f35a5b46751"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:21:44.027026 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.027003    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-model-cache" (OuterVolumeSpecName: "model-cache") pod "9506d770-6842-45fb-8489-6f35a5b46751" (UID: "9506d770-6842-45fb-8489-6f35a5b46751"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:21:44.029244 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.029213    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-dshm" (OuterVolumeSpecName: "dshm") pod "9506d770-6842-45fb-8489-6f35a5b46751" (UID: "9506d770-6842-45fb-8489-6f35a5b46751"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:21:44.029449 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.029422    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9506d770-6842-45fb-8489-6f35a5b46751-kube-api-access-qdv4c" (OuterVolumeSpecName: "kube-api-access-qdv4c") pod "9506d770-6842-45fb-8489-6f35a5b46751" (UID: "9506d770-6842-45fb-8489-6f35a5b46751"). InnerVolumeSpecName "kube-api-access-qdv4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:21:44.029569 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.029472    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9506d770-6842-45fb-8489-6f35a5b46751-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9506d770-6842-45fb-8489-6f35a5b46751" (UID: "9506d770-6842-45fb-8489-6f35a5b46751"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:21:44.030370 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.030349    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f" (OuterVolumeSpecName: "kserve-pvc-source") pod "9506d770-6842-45fb-8489-6f35a5b46751" (UID: "9506d770-6842-45fb-8489-6f35a5b46751"). InnerVolumeSpecName "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 01 21:21:44.038668 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.038642    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9506d770-6842-45fb-8489-6f35a5b46751" (UID: "9506d770-6842-45fb-8489-6f35a5b46751"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:21:44.127753 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.127707    2573 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") on node \"ip-10-0-138-219.ec2.internal\" "
Jul 01 21:21:44.127753 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.127739    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdv4c\" (UniqueName: \"kubernetes.io/projected/9506d770-6842-45fb-8489-6f35a5b46751-kube-api-access-qdv4c\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.127753 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.127749    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.127753 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.127760    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9506d770-6842-45fb-8489-6f35a5b46751-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.128174 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.127774    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.128174 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.127786    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.128174 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.127798    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9506d770-6842-45fb-8489-6f35a5b46751-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:21:44.154686 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.154665    2573 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f") on node "ip-10-0-138-219.ec2.internal"
Jul 01 21:21:44.229357 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.229277    2573 reconciler_common.go:299] "Volume detached for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\""
Jul 01 21:21:44.255036 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.255008    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"]
Jul 01 21:21:44.259568 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.259543    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp"]
Jul 01 21:21:44.938513 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.938423    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"
Jul 01 21:21:44.938513 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.938445    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb" event={"ID":"9506d770-6842-45fb-8489-6f35a5b46751","Type":"ContainerDied","Data":"e580ae700c12aaf990ccccf6f683e749bdfbba283029a59bc1b75465e1fbf92b"}
Jul 01 21:21:44.939056 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.938524    2573 scope.go:117] "RemoveContainer" containerID="954e747baac21499c4813deb9238bc8e58fc0a43ea7841e967ae84520460a03b"
Jul 01 21:21:44.957441 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.957422    2573 scope.go:117] "RemoveContainer" containerID="fb96637aff0ddf465e1f00866e113cd8f0b26c7674e27b79314793f4c872c055"
Jul 01 21:21:44.973747 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.973722    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"]
Jul 01 21:21:44.981815 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:44.981789    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb"]
Jul 01 21:21:45.591934 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:45.591906    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" path="/var/lib/kubelet/pods/4ee58395-0ba2-4058-83e4-892686bf16e5/volumes"
Jul 01 21:21:45.592311 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:45.592298    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9506d770-6842-45fb-8489-6f35a5b46751" path="/var/lib/kubelet/pods/9506d770-6842-45fb-8489-6f35a5b46751/volumes"
Jul 01 21:21:46.451803 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:46.451754    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:21:46.475023 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:46.474983    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:21:56.452482 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:56.452379    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:21:56.475306 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:56.475269    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:21:57.925401 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925364    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 01 21:21:57.925897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925697    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main"
Jul 01 21:21:57.925897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925709    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main"
Jul 01 21:21:57.925897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925720    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="llm-d-routing-sidecar"
Jul 01 21:21:57.925897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925725    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="llm-d-routing-sidecar"
Jul 01 21:21:57.925897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925735    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main"
Jul 01 21:21:57.925897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925741    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main"
Jul 01 21:21:57.925897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925790    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="main"
Jul 01 21:21:57.925897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925797    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9506d770-6842-45fb-8489-6f35a5b46751" containerName="llm-d-routing-sidecar"
Jul 01 21:21:57.925897 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.925806    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee58395-0ba2-4058-83e4-892686bf16e5" containerName="main"
Jul 01 21:21:57.928774 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.928753    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:57.931539 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.931510    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-ph46t\""
Jul 01 21:21:57.932823 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.932801    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 01 21:21:57.943102 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.943079    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 01 21:21:57.978899 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.978877    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 01 21:21:57.981071 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.981053    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:57.998036 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:57.998011    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 01 21:21:58.051415 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051381    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.051415 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051425    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgqn\" (UniqueName: \"kubernetes.io/projected/06ecda6f-0075-4799-9786-f103948f413c-kube-api-access-dxgqn\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.051675 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.051675 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.051675 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051557    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.051675 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051583    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6208b39-28fc-4309-a5f9-4887de090180-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.051966 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051683    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.051966 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051700    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.051966 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rgx\" (UniqueName: \"kubernetes.io/projected/a6208b39-28fc-4309-a5f9-4887de090180-kube-api-access-52rgx\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.051966 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051732    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.051966 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051757    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.051966 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.051790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.152303 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.152255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.152303 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.152297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgqn\" (UniqueName: \"kubernetes.io/projected/06ecda6f-0075-4799-9786-f103948f413c-kube-api-access-dxgqn\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.152547 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.152335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.152547 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.152375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.152547 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.152405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.152547 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.152441    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6208b39-28fc-4309-a5f9-4887de090180-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.152758 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.152638    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.152822 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.152767    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.152822 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.152812    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.153922 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.153452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.153922 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.153514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.153922 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.153578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rgx\" (UniqueName: \"kubernetes.io/projected/a6208b39-28fc-4309-a5f9-4887de090180-kube-api-access-52rgx\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.153922 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.153610    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.153922 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.153660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.154255 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.153998    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.155190 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.154725    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.155190 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.155128    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.156364 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.153685    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.156364 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.156034    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.162215 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.162110    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6208b39-28fc-4309-a5f9-4887de090180-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.162925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.162898    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:21:58.164392 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.164363    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgqn\" (UniqueName: \"kubernetes.io/projected/06ecda6f-0075-4799-9786-f103948f413c-kube-api-access-dxgqn\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.165363 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.164680    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:21:58.165363 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:21:58.165358    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rgx\" (UniqueName: \"kubernetes.io/projected/a6208b39-28fc-4309-a5f9-4887de090180-kube-api-access-52rgx\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:02.394873 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.394810    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:02.402313 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.402286    2573 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:02.496211 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.496171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:02.496388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.496312    2573 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:02.500784 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.500754    2573 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:02.592807 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.592768    2573 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a7984d75592a60f75acc48715216990326d59cb841472d1b5739e644ba9a7d8d/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:02.606257 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.606225    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:02.698104 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.697998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:22:02.698279 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.698138    2573 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:22:02.701825 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.701797    2573 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:22:02.715713 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.715683    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:22:02.739844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.739812    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:02.790628 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.790587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:22:02.911564 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.911522    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 01 21:22:02.961055 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:02.957838    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 01 21:22:02.964119 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:22:02.964091    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ecda6f_0075_4799_9786_f103948f413c.slice/crio-5b82b432b0ecea7a6a8bd5b5a1dedfeded7d9f10137a2416a5a5f53bd01685d6 WatchSource:0}: Error finding container 5b82b432b0ecea7a6a8bd5b5a1dedfeded7d9f10137a2416a5a5f53bd01685d6: Status 404 returned error can't find the container with id 5b82b432b0ecea7a6a8bd5b5a1dedfeded7d9f10137a2416a5a5f53bd01685d6
Jul 01 21:22:03.032840 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:03.032802    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"06ecda6f-0075-4799-9786-f103948f413c","Type":"ContainerStarted","Data":"5b82b432b0ecea7a6a8bd5b5a1dedfeded7d9f10137a2416a5a5f53bd01685d6"}
Jul 01 21:22:03.035034 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:03.035001    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"a6208b39-28fc-4309-a5f9-4887de090180","Type":"ContainerStarted","Data":"ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7"}
Jul 01 21:22:03.035150 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:03.035057    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"a6208b39-28fc-4309-a5f9-4887de090180","Type":"ContainerStarted","Data":"bc18b95cc33b82d05c820da544034bf9126d739ed45655c5566a99081daaf869"}
Jul 01 21:22:03.740796 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:03.740750    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:04.044947 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:04.044802    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"06ecda6f-0075-4799-9786-f103948f413c","Type":"ContainerStarted","Data":"6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d"}
Jul 01 21:22:04.067262 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:04.067195    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=7.067157866 podStartE2EDuration="7.067157866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:22:03.084167572 +0000 UTC m=+2155.995502198" watchObservedRunningTime="2026-07-01 21:22:04.067157866 +0000 UTC m=+2156.978492494"
Jul 01 21:22:04.069335 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:04.069287    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=7.069269588 podStartE2EDuration="7.069269588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:22:04.066758007 +0000 UTC m=+2156.978092633" watchObservedRunningTime="2026-07-01 21:22:04.069269588 +0000 UTC m=+2156.980604218"
Jul 01 21:22:06.452657 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:06.452389    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:22:06.475903 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:06.475841    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:22:12.740658 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:12.740595    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:22:12.742641 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:12.742608    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:22:16.452562 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:16.452493    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:22:16.475488 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:16.475432    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:22:22.741155 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:22.741089    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:22:26.451626 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:26.451573    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:22:26.475650 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:26.475607    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:22:32.740638 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:32.740580    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:22:36.452157 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:36.451973    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:22:36.475246 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:36.475197    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:22:42.740415 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:42.740357    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:22:46.452432 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:46.452369    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:22:46.474960 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:46.474914    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:22:52.741234 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:52.741185    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:22:56.452463 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:56.452407    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:22:56.475435 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:22:56.475390    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:23:02.740517 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:02.740471    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:23:06.451642 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:06.451597    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:23:06.475428 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:06.475392    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:23:12.740870 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:12.740810    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:23:16.451471 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:16.451430    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:23:16.475062 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:16.475024    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:23:22.740511 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:22.740464    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:23:26.452073 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:26.452024    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:23:26.475223 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:26.475185    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:23:32.741296 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:32.741197    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:23:36.452439 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:36.452389    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:23:36.475200 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:36.475156    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:23:42.741260 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:42.741221    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:23:46.452490 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:46.452439    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:23:46.475377 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:46.475342    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:23:52.740352 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:52.740306    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:23:56.451641 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:56.451590    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:23:56.475571 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:23:56.475531    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:24:02.741200 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:02.741159    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:24:06.451817 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:06.451768    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:24:06.475074 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:06.475037    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:24:12.740547 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:12.740506    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:24:16.452322 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:16.452275    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:24:16.475721 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:16.475680    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:24:22.740955 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:22.740914    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:24:26.451903 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:26.451840    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Jul 01 21:24:26.475590 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:26.475552    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 01 21:24:32.740750 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:32.740704    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:24:36.461601 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:36.461565    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:24:36.478820 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:36.478787    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:24:36.485832 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:36.485801    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:24:36.493755 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:36.493731    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:24:42.741140 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:42.741097    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:24:52.741072 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:24:52.741024    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:25:02.740594 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:25:02.740481    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:25:12.750457 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:25:12.750423    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:25:12.757974 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:25:12.757949    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:25:45.858607 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:25:45.858573    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"]
Jul 01 21:25:45.861080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:25:45.861051    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"]
Jul 01 21:26:00.881864 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:00.881794    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main" containerID="cri-o://ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf" gracePeriod=45
Jul 01 21:26:00.883245 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:00.883210    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main" containerID="cri-o://7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157" gracePeriod=45
Jul 01 21:26:01.698601 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:01.698571    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="llm-d-routing-sidecar" containerID="cri-o://5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521" gracePeriod=45
Jul 01 21:26:01.837609 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:01.837586    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:26:01.934776 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:01.934750    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:26:02.004626 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.004598    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tmp-dir\") pod \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") "
Jul 01 21:26:02.004626 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.004627    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-dshm\") pod \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") "
Jul 01 21:26:02.004818 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.004711    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhqmg\" (UniqueName: \"kubernetes.io/projected/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kube-api-access-hhqmg\") pod \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") "
Jul 01 21:26:02.004818 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.004750    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-model-cache\") pod \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") "
Jul 01 21:26:02.004818 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.004780    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kserve-provision-location\") pod \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") "
Jul 01 21:26:02.004990 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.004887    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-home\") pod \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") "
Jul 01 21:26:02.004990 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.004933    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tls-certs\") pod \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\" (UID: \"c0bfb1e1-e817-47ab-ae25-c5a71b828396\") "
Jul 01 21:26:02.005095 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.005065    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0bfb1e1-e817-47ab-ae25-c5a71b828396" (UID: "c0bfb1e1-e817-47ab-ae25-c5a71b828396"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.005322 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.005302    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.005712 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.005682    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-home" (OuterVolumeSpecName: "home") pod "c0bfb1e1-e817-47ab-ae25-c5a71b828396" (UID: "c0bfb1e1-e817-47ab-ae25-c5a71b828396"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.007174 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.007150    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kube-api-access-hhqmg" (OuterVolumeSpecName: "kube-api-access-hhqmg") pod "c0bfb1e1-e817-47ab-ae25-c5a71b828396" (UID: "c0bfb1e1-e817-47ab-ae25-c5a71b828396"). InnerVolumeSpecName "kube-api-access-hhqmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:26:02.007357 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.007331    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-dshm" (OuterVolumeSpecName: "dshm") pod "c0bfb1e1-e817-47ab-ae25-c5a71b828396" (UID: "c0bfb1e1-e817-47ab-ae25-c5a71b828396"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.007468 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.007351    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0bfb1e1-e817-47ab-ae25-c5a71b828396" (UID: "c0bfb1e1-e817-47ab-ae25-c5a71b828396"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:26:02.016946 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.016923    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0bfb1e1-e817-47ab-ae25-c5a71b828396" (UID: "c0bfb1e1-e817-47ab-ae25-c5a71b828396"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.072005 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.071953    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0bfb1e1-e817-47ab-ae25-c5a71b828396" (UID: "c0bfb1e1-e817-47ab-ae25-c5a71b828396"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.106298 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106272    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-kserve-provision-location\") pod \"ab193660-208e-4e20-a524-a7abb9a93f6c\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") "
Jul 01 21:26:02.106421 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106315    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp5qv\" (UniqueName: \"kubernetes.io/projected/ab193660-208e-4e20-a524-a7abb9a93f6c-kube-api-access-qp5qv\") pod \"ab193660-208e-4e20-a524-a7abb9a93f6c\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") "
Jul 01 21:26:02.106421 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106339    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-model-cache\") pod \"ab193660-208e-4e20-a524-a7abb9a93f6c\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") "
Jul 01 21:26:02.106421 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106370    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab193660-208e-4e20-a524-a7abb9a93f6c-tls-certs\") pod \"ab193660-208e-4e20-a524-a7abb9a93f6c\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") "
Jul 01 21:26:02.106421 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106416    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-dshm\") pod \"ab193660-208e-4e20-a524-a7abb9a93f6c\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") "
Jul 01 21:26:02.106636 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106437    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-home\") pod \"ab193660-208e-4e20-a524-a7abb9a93f6c\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") "
Jul 01 21:26:02.106636 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106471    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-tmp-dir\") pod \"ab193660-208e-4e20-a524-a7abb9a93f6c\" (UID: \"ab193660-208e-4e20-a524-a7abb9a93f6c\") "
Jul 01 21:26:02.106636 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106582    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab193660-208e-4e20-a524-a7abb9a93f6c" (UID: "ab193660-208e-4e20-a524-a7abb9a93f6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.106788 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106755    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.106788 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106773    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.106914 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106788    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.106914 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106800    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.106914 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106813    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.106914 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106827    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhqmg\" (UniqueName: \"kubernetes.io/projected/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kube-api-access-hhqmg\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.106914 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.106840    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0bfb1e1-e817-47ab-ae25-c5a71b828396-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.107102 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.107059    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-home" (OuterVolumeSpecName: "home") pod "ab193660-208e-4e20-a524-a7abb9a93f6c" (UID: "ab193660-208e-4e20-a524-a7abb9a93f6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.108747 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.108728    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-dshm" (OuterVolumeSpecName: "dshm") pod "ab193660-208e-4e20-a524-a7abb9a93f6c" (UID: "ab193660-208e-4e20-a524-a7abb9a93f6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.108950 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.108926    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab193660-208e-4e20-a524-a7abb9a93f6c-kube-api-access-qp5qv" (OuterVolumeSpecName: "kube-api-access-qp5qv") pod "ab193660-208e-4e20-a524-a7abb9a93f6c" (UID: "ab193660-208e-4e20-a524-a7abb9a93f6c"). InnerVolumeSpecName "kube-api-access-qp5qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:26:02.108950 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.108933    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab193660-208e-4e20-a524-a7abb9a93f6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab193660-208e-4e20-a524-a7abb9a93f6c" (UID: "ab193660-208e-4e20-a524-a7abb9a93f6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:26:02.125602 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.125579    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab193660-208e-4e20-a524-a7abb9a93f6c" (UID: "ab193660-208e-4e20-a524-a7abb9a93f6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.177974 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.177910    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab193660-208e-4e20-a524-a7abb9a93f6c" (UID: "ab193660-208e-4e20-a524-a7abb9a93f6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:26:02.207984 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.207960    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.207984 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.207981    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5qv\" (UniqueName: \"kubernetes.io/projected/ab193660-208e-4e20-a524-a7abb9a93f6c-kube-api-access-qp5qv\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.208117 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.207990    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab193660-208e-4e20-a524-a7abb9a93f6c-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.208117 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.207999    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.208117 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.208006    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.208117 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.208014    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab193660-208e-4e20-a524-a7abb9a93f6c-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:26:02.262123 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.262099    2573 generic.go:356] "Generic (PLEG): container finished" podID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerID="ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf" exitCode=0
Jul 01 21:26:02.262123 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.262120    2573 generic.go:356] "Generic (PLEG): container finished" podID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerID="5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521" exitCode=0
Jul 01 21:26:02.262269 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.262165    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"
Jul 01 21:26:02.262269 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.262179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" event={"ID":"ab193660-208e-4e20-a524-a7abb9a93f6c","Type":"ContainerDied","Data":"ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf"}
Jul 01 21:26:02.262363 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.262276    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" event={"ID":"ab193660-208e-4e20-a524-a7abb9a93f6c","Type":"ContainerDied","Data":"5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521"}
Jul 01 21:26:02.262363 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.262306    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2" event={"ID":"ab193660-208e-4e20-a524-a7abb9a93f6c","Type":"ContainerDied","Data":"5a753a624105b8644389640923a1e5abf38d446d7e196634624d7ef4a78d1f97"}
Jul 01 21:26:02.262363 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.262333    2573 scope.go:117] "RemoveContainer" containerID="ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf"
Jul 01 21:26:02.263812 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.263782    2573 generic.go:356] "Generic (PLEG): container finished" podID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerID="7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157" exitCode=0
Jul 01 21:26:02.263931 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.263822    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" event={"ID":"c0bfb1e1-e817-47ab-ae25-c5a71b828396","Type":"ContainerDied","Data":"7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157"}
Jul 01 21:26:02.263931 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.263888    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"
Jul 01 21:26:02.264034 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.263953    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8" event={"ID":"c0bfb1e1-e817-47ab-ae25-c5a71b828396","Type":"ContainerDied","Data":"631dcccfd95506a92af8d1d01b5ac3d4fe37be1e6af3284ea2e2754c6221dbd0"}
Jul 01 21:26:02.272465 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.272450    2573 scope.go:117] "RemoveContainer" containerID="5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a"
Jul 01 21:26:02.281034 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.281016    2573 scope.go:117] "RemoveContainer" containerID="5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521"
Jul 01 21:26:02.286674 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.286651    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"]
Jul 01 21:26:02.289656 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.289500    2573 scope.go:117] "RemoveContainer" containerID="ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf"
Jul 01 21:26:02.289804 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:26:02.289784    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf\": container with ID starting with ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf not found: ID does not exist" containerID="ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf"
Jul 01 21:26:02.289894 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.289810    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf"} err="failed to get container status \"ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf\": rpc error: code = NotFound desc = could not find container \"ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf\": container with ID starting with ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf not found: ID does not exist"
Jul 01 21:26:02.289894 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.289830    2573 scope.go:117] "RemoveContainer" containerID="5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a"
Jul 01 21:26:02.290138 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:26:02.290121    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a\": container with ID starting with 5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a not found: ID does not exist" containerID="5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a"
Jul 01 21:26:02.290199 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.290142    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a"} err="failed to get container status \"5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a\": rpc error: code = NotFound desc = could not find container \"5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a\": container with ID starting with 5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a not found: ID does not exist"
Jul 01 21:26:02.290199 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.290157    2573 scope.go:117] "RemoveContainer" containerID="5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521"
Jul 01 21:26:02.290383 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:26:02.290366    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521\": container with ID starting with 5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521 not found: ID does not exist" containerID="5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521"
Jul 01 21:26:02.290440 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.290387    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521"} err="failed to get container status \"5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521\": rpc error: code = NotFound desc = could not find container \"5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521\": container with ID starting with 5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521 not found: ID does not exist"
Jul 01 21:26:02.290440 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.290401    2573 scope.go:117] "RemoveContainer" containerID="ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf"
Jul 01 21:26:02.290620 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.290604    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf"} err="failed to get container status \"ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf\": rpc error: code = NotFound desc = could not find container \"ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf\": container with ID starting with ce167cee8e420ab1b84b2ac30e6c2a5746ff9cffff83f8ce042f8d6a74f347bf not found: ID does not exist"
Jul 01 21:26:02.290715 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.290620    2573 scope.go:117] "RemoveContainer" containerID="5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a"
Jul 01 21:26:02.290839 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.290819    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a"} err="failed to get container status \"5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a\": rpc error: code = NotFound desc = could not find container \"5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a\": container with ID starting with 5a674ea39a0ad21abe613eda81656130b2d5621e2a8962abb0e1ae35e3d8669a not found: ID does not exist"
Jul 01 21:26:02.290839 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.290838    2573 scope.go:117] "RemoveContainer" containerID="5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521"
Jul 01 21:26:02.291037 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.290884    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2"]
Jul 01 21:26:02.291094 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.291068    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521"} err="failed to get container status \"5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521\": rpc error: code = NotFound desc = could not find container \"5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521\": container with ID starting with 5b3d5286a9290d87617406872cfef3005e0ccf96bf5719f12f2fe614a1dc6521 not found: ID does not exist"
Jul 01 21:26:02.291094 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.291084    2573 scope.go:117] "RemoveContainer" containerID="7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157"
Jul 01 21:26:02.300365 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.300348    2573 scope.go:117] "RemoveContainer" containerID="ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf"
Jul 01 21:26:02.302332 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.302302    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"]
Jul 01 21:26:02.306148 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.306130    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8"]
Jul 01 21:26:02.308697 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.308683    2573 scope.go:117] "RemoveContainer" containerID="7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157"
Jul 01 21:26:02.308966 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:26:02.308936    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157\": container with ID starting with 7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157 not found: ID does not exist" containerID="7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157"
Jul 01 21:26:02.309054 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.308974    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157"} err="failed to get container status \"7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157\": rpc error: code = NotFound desc = could not find container \"7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157\": container with ID starting with 7b5de4f1700c86e61db691cef03abfe8cd63e9e327f9436a2b9de05f5e1cb157 not found: ID does not exist"
Jul 01 21:26:02.309054 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.308996    2573 scope.go:117] "RemoveContainer" containerID="ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf"
Jul 01 21:26:02.309286 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:26:02.309269    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf\": container with ID starting with ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf not found: ID does not exist" containerID="ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf"
Jul 01 21:26:02.309333 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:02.309291    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf"} err="failed to get container status \"ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf\": rpc error: code = NotFound desc = could not find container \"ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf\": container with ID starting with ebf9fff9ecfbf6cfdfc84ef2ecf3c6a7fdfbd5625d2bb912edbda6e8ee66c4cf not found: ID does not exist"
Jul 01 21:26:03.591898 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:03.591848    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" path="/var/lib/kubelet/pods/ab193660-208e-4e20-a524-a7abb9a93f6c/volumes"
Jul 01 21:26:03.592510 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:03.592489    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" path="/var/lib/kubelet/pods/c0bfb1e1-e817-47ab-ae25-c5a71b828396/volumes"
Jul 01 21:26:07.731907 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:07.731876    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:26:07.735497 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:07.735472    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:26:07.741881 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:07.741836    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:26:07.744704 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:07.744687    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:26:31.225656 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225582    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"]
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225908    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="storage-initializer"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225920    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="storage-initializer"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225931    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225936    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225947    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225954    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225967    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="llm-d-routing-sidecar"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225972    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="llm-d-routing-sidecar"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225977    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="storage-initializer"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.225982    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="storage-initializer"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.226031    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="main"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.226041    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bfb1e1-e817-47ab-ae25-c5a71b828396" containerName="main"
Jul 01 21:26:31.226080 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.226046    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab193660-208e-4e20-a524-a7abb9a93f6c" containerName="llm-d-routing-sidecar"
Jul 01 21:26:31.229294 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.229277    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.232188 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.232167    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:26:31.232320 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.232167    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-r49z4\""
Jul 01 21:26:31.232320 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.232189    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 01 21:26:31.243974 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.243951    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"]
Jul 01 21:26:31.251799 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.251775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tmp-dir\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.251943 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.251809    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-dshm\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.251943 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.251844    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-home\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.252068 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.251926    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brp4h\" (UniqueName: \"kubernetes.io/projected/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kube-api-access-brp4h\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.252068 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.252001    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-model-cache\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.252068 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.252042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.252230 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.252095    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tls-certs\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.263902 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.263875    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"]
Jul 01 21:26:31.267122 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.267107    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.281501 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.281483    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"]
Jul 01 21:26:31.352609 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352586    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-home\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.352738 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352618    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.352738 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-home\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.352738 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brp4h\" (UniqueName: \"kubernetes.io/projected/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kube-api-access-brp4h\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.352738 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352711    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.352958 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352751    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-model-cache\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.352958 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldmk\" (UniqueName: \"kubernetes.io/projected/9695d6e5-2cba-462f-9194-0bf2e312e347-kube-api-access-2ldmk\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.352958 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.352958 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.353117 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352963    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tls-certs\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.353117 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.353038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tmp-dir\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.353117 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.352974    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-home\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.353117 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.353077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-dshm\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.353341 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.353157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.353341 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.353097    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-model-cache\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.353341 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.353186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9695d6e5-2cba-462f-9194-0bf2e312e347-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.353341 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.353234    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.353528 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.353429    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tmp-dir\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.355106 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.355089    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-dshm\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.355272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.355256    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tls-certs\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.360420 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.360397    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brp4h\" (UniqueName: \"kubernetes.io/projected/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kube-api-access-brp4h\") pod \"router-with-refs-pd-test-kserve-88fcd5785-4gknp\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.454156 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454125    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454161    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-home\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454194    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454231    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldmk\" (UniqueName: \"kubernetes.io/projected/9695d6e5-2cba-462f-9194-0bf2e312e347-kube-api-access-2ldmk\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454530 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454530 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9695d6e5-2cba-462f-9194-0bf2e312e347-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454635 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454615    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454697 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454658    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454752 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454724    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.454876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.454843    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-home\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.456651 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.456630    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.456728 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.456703    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9695d6e5-2cba-462f-9194-0bf2e312e347-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.462363 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.462342    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldmk\" (UniqueName: \"kubernetes.io/projected/9695d6e5-2cba-462f-9194-0bf2e312e347-kube-api-access-2ldmk\") pod \"router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.538460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.538382    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:31.577032 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.576999    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:31.686515 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.686488    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"]
Jul 01 21:26:31.688777 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:26:31.688750    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27e8da6_a33a_4a8c_bf54_739f9221f4f6.slice/crio-9b63a0cc731d32506587fa2e0215f6a3a161765278d6c8b9b401db234ccc4437 WatchSource:0}: Error finding container 9b63a0cc731d32506587fa2e0215f6a3a161765278d6c8b9b401db234ccc4437: Status 404 returned error can't find the container with id 9b63a0cc731d32506587fa2e0215f6a3a161765278d6c8b9b401db234ccc4437
Jul 01 21:26:31.725123 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:31.725095    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"]
Jul 01 21:26:31.728460 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:26:31.728433    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9695d6e5_2cba_462f_9194_0bf2e312e347.slice/crio-1cd9f694fb87ca7a878188a813e5787f2e885607d8e6e188a1583f0c34f6846c WatchSource:0}: Error finding container 1cd9f694fb87ca7a878188a813e5787f2e885607d8e6e188a1583f0c34f6846c: Status 404 returned error can't find the container with id 1cd9f694fb87ca7a878188a813e5787f2e885607d8e6e188a1583f0c34f6846c
Jul 01 21:26:32.399845 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:32.399811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" event={"ID":"f27e8da6-a33a-4a8c-bf54-739f9221f4f6","Type":"ContainerStarted","Data":"3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054"}
Jul 01 21:26:32.400285 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:32.399918    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" event={"ID":"f27e8da6-a33a-4a8c-bf54-739f9221f4f6","Type":"ContainerStarted","Data":"9b63a0cc731d32506587fa2e0215f6a3a161765278d6c8b9b401db234ccc4437"}
Jul 01 21:26:32.400285 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:32.400041    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:32.402036 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:32.402008    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" event={"ID":"9695d6e5-2cba-462f-9194-0bf2e312e347","Type":"ContainerStarted","Data":"cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3"}
Jul 01 21:26:32.402154 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:32.402059    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" event={"ID":"9695d6e5-2cba-462f-9194-0bf2e312e347","Type":"ContainerStarted","Data":"1cd9f694fb87ca7a878188a813e5787f2e885607d8e6e188a1583f0c34f6846c"}
Jul 01 21:26:33.408910 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:33.408839    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" event={"ID":"f27e8da6-a33a-4a8c-bf54-739f9221f4f6","Type":"ContainerStarted","Data":"d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd"}
Jul 01 21:26:35.420546 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:35.420515    2573 generic.go:356] "Generic (PLEG): container finished" podID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerID="cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3" exitCode=0
Jul 01 21:26:35.421031 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:35.420570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" event={"ID":"9695d6e5-2cba-462f-9194-0bf2e312e347","Type":"ContainerDied","Data":"cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3"}
Jul 01 21:26:36.428438 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:36.428388    2573 generic.go:356] "Generic (PLEG): container finished" podID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerID="d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd" exitCode=0
Jul 01 21:26:36.428872 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:36.428463    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" event={"ID":"f27e8da6-a33a-4a8c-bf54-739f9221f4f6","Type":"ContainerDied","Data":"d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd"}
Jul 01 21:26:36.430610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:36.430587    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" event={"ID":"9695d6e5-2cba-462f-9194-0bf2e312e347","Type":"ContainerStarted","Data":"f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722"}
Jul 01 21:26:36.472580 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:36.472524    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podStartSLOduration=1.871345216 podStartE2EDuration="5.472509798s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.601164582s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:26:36.470623602 +0000 UTC m=+2429.381958230" watchObservedRunningTime="2026-07-01 21:26:36.472509798 +0000 UTC m=+2429.383844424"
Jul 01 21:26:37.438937 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:37.438898    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" event={"ID":"f27e8da6-a33a-4a8c-bf54-739f9221f4f6","Type":"ContainerStarted","Data":"efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3"}
Jul 01 21:26:37.465837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:37.465781    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podStartSLOduration=3.062831321 podStartE2EDuration="6.465760926s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.402929605s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:26:37.463138564 +0000 UTC m=+2430.374473204" watchObservedRunningTime="2026-07-01 21:26:37.465760926 +0000 UTC m=+2430.377095553"
Jul 01 21:26:41.539412 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:41.539371    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:41.539907 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:41.539475    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:41.540812 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:41.540771    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:26:41.559099 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:41.559073    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:26:41.577472 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:41.577450    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:41.577607 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:41.577568    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:26:41.579101 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:41.579074    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:26:51.539505 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:51.539455    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:26:51.577716 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:26:51.577676    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:27:01.539579 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:01.539531    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:27:01.578012 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:01.577975    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:27:11.539431 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:11.539382    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:27:11.577400 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:11.577357    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:27:21.539819 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:21.539768    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:27:21.577497 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:21.577454    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:27:31.539413 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:31.539365    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:27:31.578260 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:31.578214    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:27:41.539633 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:41.539567    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:27:41.578366 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:41.578324    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:27:51.539345 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:51.539299    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:27:51.577486 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:27:51.577448    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:28:01.539360 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:01.539219    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:28:01.578074 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:01.578034    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:28:01.635632 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:01.635597    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 01 21:28:11.539650 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:11.539595    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:28:11.577795 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:11.577751    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:28:12.585435 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:12.585402    2573 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-ph46t\" not found"
Jul 01 21:28:12.652014 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:12.651981    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:12.652182 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:12.652056    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs podName:06ecda6f-0075-4799-9786-f103948f413c nodeName:}" failed. No retries permitted until 2026-07-01 21:28:13.152043045 +0000 UTC m=+2526.063377650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "06ecda6f-0075-4799-9786-f103948f413c") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:13.157452 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:13.157418    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:13.157624 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:13.157494    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs podName:06ecda6f-0075-4799-9786-f103948f413c nodeName:}" failed. No retries permitted until 2026-07-01 21:28:14.15747662 +0000 UTC m=+2527.068811236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "06ecda6f-0075-4799-9786-f103948f413c") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:14.167254 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:14.167220    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:14.167631 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:14.167298    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs podName:06ecda6f-0075-4799-9786-f103948f413c nodeName:}" failed. No retries permitted until 2026-07-01 21:28:16.167280401 +0000 UTC m=+2529.078615005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "06ecda6f-0075-4799-9786-f103948f413c") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:16.187496 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:16.187461    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:16.187909 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:16.187538    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs podName:06ecda6f-0075-4799-9786-f103948f413c nodeName:}" failed. No retries permitted until 2026-07-01 21:28:20.187524754 +0000 UTC m=+2533.098859358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "06ecda6f-0075-4799-9786-f103948f413c") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:16.658785 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:16.658719    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" containerID="cri-o://ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7" gracePeriod=45
Jul 01 21:28:16.751811 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:16.751767    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 01 21:28:17.830192 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.830169    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:28:17.904609 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.904544    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-model-cache\") pod \"a6208b39-28fc-4309-a5f9-4887de090180\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") "
Jul 01 21:28:17.904765 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.904616    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-dshm\") pod \"a6208b39-28fc-4309-a5f9-4887de090180\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") "
Jul 01 21:28:17.904765 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.904649    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rgx\" (UniqueName: \"kubernetes.io/projected/a6208b39-28fc-4309-a5f9-4887de090180-kube-api-access-52rgx\") pod \"a6208b39-28fc-4309-a5f9-4887de090180\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") "
Jul 01 21:28:17.904765 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.904664    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-tmp-dir\") pod \"a6208b39-28fc-4309-a5f9-4887de090180\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") "
Jul 01 21:28:17.904971 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.904766    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"a6208b39-28fc-4309-a5f9-4887de090180\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") "
Jul 01 21:28:17.904971 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.904815    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-home\") pod \"a6208b39-28fc-4309-a5f9-4887de090180\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") "
Jul 01 21:28:17.904971 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.904872    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6208b39-28fc-4309-a5f9-4887de090180-tls-certs\") pod \"a6208b39-28fc-4309-a5f9-4887de090180\" (UID: \"a6208b39-28fc-4309-a5f9-4887de090180\") "
Jul 01 21:28:17.905141 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.904843    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6208b39-28fc-4309-a5f9-4887de090180" (UID: "a6208b39-28fc-4309-a5f9-4887de090180"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:28:17.905435 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.905239    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:17.905536 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.905454    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-home" (OuterVolumeSpecName: "home") pod "a6208b39-28fc-4309-a5f9-4887de090180" (UID: "a6208b39-28fc-4309-a5f9-4887de090180"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:28:17.907457 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.907419    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6208b39-28fc-4309-a5f9-4887de090180-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6208b39-28fc-4309-a5f9-4887de090180" (UID: "a6208b39-28fc-4309-a5f9-4887de090180"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:28:17.907587 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.907523    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6208b39-28fc-4309-a5f9-4887de090180-kube-api-access-52rgx" (OuterVolumeSpecName: "kube-api-access-52rgx") pod "a6208b39-28fc-4309-a5f9-4887de090180" (UID: "a6208b39-28fc-4309-a5f9-4887de090180"). InnerVolumeSpecName "kube-api-access-52rgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:28:17.907587 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.907570    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-dshm" (OuterVolumeSpecName: "dshm") pod "a6208b39-28fc-4309-a5f9-4887de090180" (UID: "a6208b39-28fc-4309-a5f9-4887de090180"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:28:17.907785 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.907766    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f" (OuterVolumeSpecName: "kserve-pvc-source") pod "a6208b39-28fc-4309-a5f9-4887de090180" (UID: "a6208b39-28fc-4309-a5f9-4887de090180"). InnerVolumeSpecName "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 01 21:28:17.916520 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.916497    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6208b39-28fc-4309-a5f9-4887de090180" (UID: "a6208b39-28fc-4309-a5f9-4887de090180"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:28:17.961366 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.961338    2573 generic.go:356] "Generic (PLEG): container finished" podID="a6208b39-28fc-4309-a5f9-4887de090180" containerID="ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7" exitCode=0
Jul 01 21:28:17.961486 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.961393    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 01 21:28:17.961486 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.961431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"a6208b39-28fc-4309-a5f9-4887de090180","Type":"ContainerDied","Data":"ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7"}
Jul 01 21:28:17.961486 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.961482    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"a6208b39-28fc-4309-a5f9-4887de090180","Type":"ContainerDied","Data":"bc18b95cc33b82d05c820da544034bf9126d739ed45655c5566a99081daaf869"}
Jul 01 21:28:17.961590 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.961504    2573 scope.go:117] "RemoveContainer" containerID="ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7"
Jul 01 21:28:17.970298 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.970278    2573 scope.go:117] "RemoveContainer" containerID="ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7"
Jul 01 21:28:17.970591 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:17.970573    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7\": container with ID starting with ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7 not found: ID does not exist" containerID="ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7"
Jul 01 21:28:17.970656 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.970602    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7"} err="failed to get container status \"ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7\": rpc error: code = NotFound desc = could not find container \"ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7\": container with ID starting with ced4dc73b6971b2c6137a1e49861b4782bbc3c626bed783400f34aaf73c844d7 not found: ID does not exist"
Jul 01 21:28:17.985637 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.985612    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 01 21:28:17.988196 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:17.988175    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 01 21:28:18.004531 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:18.004501    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 01 21:28:18.006547 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:18.006526    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:18.006547 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:18.006548    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rgx\" (UniqueName: \"kubernetes.io/projected/a6208b39-28fc-4309-a5f9-4887de090180-kube-api-access-52rgx\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:18.006679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:18.006558    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:18.006679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:18.006569    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6208b39-28fc-4309-a5f9-4887de090180-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:18.006679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:18.006577    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6208b39-28fc-4309-a5f9-4887de090180-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:18.752046 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:18.751989    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded"
Jul 01 21:28:19.591111 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:19.591079    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6208b39-28fc-4309-a5f9-4887de090180" path="/var/lib/kubelet/pods/a6208b39-28fc-4309-a5f9-4887de090180/volumes"
Jul 01 21:28:20.232031 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:20.231987    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:20.232220 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:20.232072    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs podName:06ecda6f-0075-4799-9786-f103948f413c nodeName:}" failed. No retries permitted until 2026-07-01 21:28:28.232056242 +0000 UTC m=+2541.143390851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "06ecda6f-0075-4799-9786-f103948f413c") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:21.539899 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:21.539830    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:28:21.578183 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:21.578142    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:28:28.305589 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:28.305553    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:28.306078 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:28.305635    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs podName:06ecda6f-0075-4799-9786-f103948f413c nodeName:}" failed. No retries permitted until 2026-07-01 21:28:44.305617758 +0000 UTC m=+2557.216952370 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "06ecda6f-0075-4799-9786-f103948f413c") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 01 21:28:31.539144 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:31.539099    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:28:31.578811 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:31.578765    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:28:33.027139 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:33.027038    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="06ecda6f-0075-4799-9786-f103948f413c" containerName="main" containerID="cri-o://6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d" gracePeriod=45
Jul 01 21:28:34.049037 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:34.049004    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ecda6f_0075_4799_9786_f103948f413c.slice/crio-6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d.scope\": RecentStats: unable to find data in memory cache]"
Jul 01 21:28:34.193362 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.193339    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:28:34.259366 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.259337    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-dshm\") pod \"06ecda6f-0075-4799-9786-f103948f413c\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") "
Jul 01 21:28:34.259366 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.259365    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-tmp-dir\") pod \"06ecda6f-0075-4799-9786-f103948f413c\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") "
Jul 01 21:28:34.259591 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.259448    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") pod \"06ecda6f-0075-4799-9786-f103948f413c\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") "
Jul 01 21:28:34.259591 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.259488    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs\") pod \"06ecda6f-0075-4799-9786-f103948f413c\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") "
Jul 01 21:28:34.259591 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.259540    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxgqn\" (UniqueName: \"kubernetes.io/projected/06ecda6f-0075-4799-9786-f103948f413c-kube-api-access-dxgqn\") pod \"06ecda6f-0075-4799-9786-f103948f413c\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") "
Jul 01 21:28:34.259591 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.259575    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-model-cache\") pod \"06ecda6f-0075-4799-9786-f103948f413c\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") "
Jul 01 21:28:34.259879 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.259643    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-home\") pod \"06ecda6f-0075-4799-9786-f103948f413c\" (UID: \"06ecda6f-0075-4799-9786-f103948f413c\") "
Jul 01 21:28:34.260298 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.260192    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-model-cache" (OuterVolumeSpecName: "model-cache") pod "06ecda6f-0075-4799-9786-f103948f413c" (UID: "06ecda6f-0075-4799-9786-f103948f413c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:28:34.260787 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.260444    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-home" (OuterVolumeSpecName: "home") pod "06ecda6f-0075-4799-9786-f103948f413c" (UID: "06ecda6f-0075-4799-9786-f103948f413c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:28:34.262222 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.262186    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06ecda6f-0075-4799-9786-f103948f413c" (UID: "06ecda6f-0075-4799-9786-f103948f413c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:28:34.262435 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.262403    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-dshm" (OuterVolumeSpecName: "dshm") pod "06ecda6f-0075-4799-9786-f103948f413c" (UID: "06ecda6f-0075-4799-9786-f103948f413c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:28:34.262435 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.262423    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ecda6f-0075-4799-9786-f103948f413c-kube-api-access-dxgqn" (OuterVolumeSpecName: "kube-api-access-dxgqn") pod "06ecda6f-0075-4799-9786-f103948f413c" (UID: "06ecda6f-0075-4799-9786-f103948f413c"). InnerVolumeSpecName "kube-api-access-dxgqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:28:34.262974 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.262950    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f" (OuterVolumeSpecName: "kserve-pvc-source") pod "06ecda6f-0075-4799-9786-f103948f413c" (UID: "06ecda6f-0075-4799-9786-f103948f413c"). InnerVolumeSpecName "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 01 21:28:34.271753 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.271732    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06ecda6f-0075-4799-9786-f103948f413c" (UID: "06ecda6f-0075-4799-9786-f103948f413c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:28:34.360330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.360301    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:34.360330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.360324    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:34.360330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.360333    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:34.360527 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.360363    2573 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") on node \"ip-10-0-138-219.ec2.internal\" "
Jul 01 21:28:34.360527 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.360373    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ecda6f-0075-4799-9786-f103948f413c-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:34.360527 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.360382    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxgqn\" (UniqueName: \"kubernetes.io/projected/06ecda6f-0075-4799-9786-f103948f413c-kube-api-access-dxgqn\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:34.360527 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.360390    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ecda6f-0075-4799-9786-f103948f413c-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:28:34.387627 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.387610    2573 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f") on node "ip-10-0-138-219.ec2.internal"
Jul 01 21:28:34.461272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:34.461249    2573 reconciler_common.go:299] "Volume detached for volume \"pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed28fde6d0ee547f\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"csi-7d3ee060a06e303c98afb3cfb425b71fc4b75b9b2ed96e78393bbf15f93f36e5\""
Jul 01 21:28:35.046441 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.046412    2573 generic.go:356] "Generic (PLEG): container finished" podID="06ecda6f-0075-4799-9786-f103948f413c" containerID="6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d" exitCode=0
Jul 01 21:28:35.046613 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.046465    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 01 21:28:35.046613 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.046499    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"06ecda6f-0075-4799-9786-f103948f413c","Type":"ContainerDied","Data":"6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d"}
Jul 01 21:28:35.046613 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.046555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"06ecda6f-0075-4799-9786-f103948f413c","Type":"ContainerDied","Data":"5b82b432b0ecea7a6a8bd5b5a1dedfeded7d9f10137a2416a5a5f53bd01685d6"}
Jul 01 21:28:35.046613 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.046576    2573 scope.go:117] "RemoveContainer" containerID="6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d"
Jul 01 21:28:35.056055 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.055738    2573 scope.go:117] "RemoveContainer" containerID="6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d"
Jul 01 21:28:35.056055 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:28:35.055998    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d\": container with ID starting with 6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d not found: ID does not exist" containerID="6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d"
Jul 01 21:28:35.056055 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.056019    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d"} err="failed to get container status \"6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d\": rpc error: code = NotFound desc = could not find container \"6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d\": container with ID starting with 6e9140a87870b16c4d8f733ed653796dcea74fce137b4bdeae17271e8185ce2d not found: ID does not exist"
Jul 01 21:28:35.068537 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.068513    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 01 21:28:35.072092 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.072066    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 01 21:28:35.591476 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:35.591436    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06ecda6f-0075-4799-9786-f103948f413c" path="/var/lib/kubelet/pods/06ecda6f-0075-4799-9786-f103948f413c/volumes"
Jul 01 21:28:41.539359 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:41.539310    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:28:41.578250 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:41.578207    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:28:45.192083 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.192036    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"]
Jul 01 21:28:45.192679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.192387    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main"
Jul 01 21:28:45.192679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.192399    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main"
Jul 01 21:28:45.192679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.192407    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ecda6f-0075-4799-9786-f103948f413c" containerName="main"
Jul 01 21:28:45.192679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.192412    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ecda6f-0075-4799-9786-f103948f413c" containerName="main"
Jul 01 21:28:45.192679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.192468    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ecda6f-0075-4799-9786-f103948f413c" containerName="main"
Jul 01 21:28:45.192679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.192478    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6208b39-28fc-4309-a5f9-4887de090180" containerName="main"
Jul 01 21:28:45.194583 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.194565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.197346 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.197325    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 01 21:28:45.210124 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.210099    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"]
Jul 01 21:28:45.247030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.246993    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-model-cache\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.247030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.247024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kserve-provision-location\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.247233 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.247047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tmp-dir\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.247233 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.247140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-home\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.247233 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.247175    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-dshm\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.247233 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.247210    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84z62\" (UniqueName: \"kubernetes.io/projected/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kube-api-access-84z62\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.247463 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.247249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tls-certs\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.348452 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.348413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tmp-dir\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.348452 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.348450    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-home\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.348701 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.348473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-dshm\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.348701 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.348497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84z62\" (UniqueName: \"kubernetes.io/projected/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kube-api-access-84z62\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.348701 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.348537    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tls-certs\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.348701 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.348581    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-model-cache\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.348978 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.348881    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-home\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.348978 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.348909    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tmp-dir\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.349095 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.348986    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-model-cache\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.349095 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.349021    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kserve-provision-location\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.349332 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.349310    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kserve-provision-location\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.350879 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.350840    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-dshm\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.351056 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.351038    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tls-certs\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.356692 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.356668    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84z62\" (UniqueName: \"kubernetes.io/projected/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kube-api-access-84z62\") pod \"stop-feature-test-kserve-8486474f84-czrmh\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.504781 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.504700    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:45.642290 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:45.642235    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"]
Jul 01 21:28:45.645247 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:28:45.645208    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce1e791_4cc1_4780_8844_7a2f9f4aea15.slice/crio-7770ed1235c67cfee8fcc041018ed205b78cd0b9640a67f187172e804e1c5158 WatchSource:0}: Error finding container 7770ed1235c67cfee8fcc041018ed205b78cd0b9640a67f187172e804e1c5158: Status 404 returned error can't find the container with id 7770ed1235c67cfee8fcc041018ed205b78cd0b9640a67f187172e804e1c5158
Jul 01 21:28:46.101389 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:46.101355    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" event={"ID":"8ce1e791-4cc1-4780-8844-7a2f9f4aea15","Type":"ContainerStarted","Data":"faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb"}
Jul 01 21:28:46.101550 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:46.101406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" event={"ID":"8ce1e791-4cc1-4780-8844-7a2f9f4aea15","Type":"ContainerStarted","Data":"7770ed1235c67cfee8fcc041018ed205b78cd0b9640a67f187172e804e1c5158"}
Jul 01 21:28:50.122990 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:50.122954    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerID="faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb" exitCode=0
Jul 01 21:28:50.123367 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:50.123025    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" event={"ID":"8ce1e791-4cc1-4780-8844-7a2f9f4aea15","Type":"ContainerDied","Data":"faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb"}
Jul 01 21:28:51.130476 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:51.130441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" event={"ID":"8ce1e791-4cc1-4780-8844-7a2f9f4aea15","Type":"ContainerStarted","Data":"e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521"}
Jul 01 21:28:51.154776 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:51.154726    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podStartSLOduration=2.4374958319999998 podStartE2EDuration="6.154710044s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717214212s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:28:51.152004727 +0000 UTC m=+2564.063339353" watchObservedRunningTime="2026-07-01 21:28:51.154710044 +0000 UTC m=+2564.066044670"
Jul 01 21:28:51.505065 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:51.504980    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:51.539238 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:51.539181    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:28:51.578335 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:51.578284    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:28:55.505893 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:55.505832    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:28:55.506970 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:28:55.506936    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:29:01.539380 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:01.539330    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8001/health\": dial tcp 10.132.0.54:8001: connect: connection refused"
Jul 01 21:29:01.577476 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:01.577443    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Jul 01 21:29:05.505331 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:05.505287    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:29:11.549323 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:11.549237    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:29:11.562121 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:11.562097    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:29:11.592244 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:11.592221    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:29:11.601151 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:11.601125    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:29:15.505982 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:15.505935    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:29:20.978654 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:20.978607    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"]
Jul 01 21:29:20.984956 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:20.984928    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"]
Jul 01 21:29:25.505328 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:25.505285    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:29:35.505753 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:35.505712    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:29:36.007580 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:36.007540    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main" containerID="cri-o://efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3" gracePeriod=45
Jul 01 21:29:36.013992 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:36.013959    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main" containerID="cri-o://f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722" gracePeriod=45
Jul 01 21:29:36.824540 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:36.824467    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="llm-d-routing-sidecar" containerID="cri-o://3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054" gracePeriod=45
Jul 01 21:29:36.965082 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:36.965055    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:29:37.009891 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.009850    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-home\") pod \"9695d6e5-2cba-462f-9194-0bf2e312e347\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") "
Jul 01 21:29:37.010014 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.009912    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-model-cache\") pod \"9695d6e5-2cba-462f-9194-0bf2e312e347\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") "
Jul 01 21:29:37.010014 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.009955    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9695d6e5-2cba-462f-9194-0bf2e312e347-tls-certs\") pod \"9695d6e5-2cba-462f-9194-0bf2e312e347\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") "
Jul 01 21:29:37.010014 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.009998    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-dshm\") pod \"9695d6e5-2cba-462f-9194-0bf2e312e347\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") "
Jul 01 21:29:37.010170 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.010028    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-kserve-provision-location\") pod \"9695d6e5-2cba-462f-9194-0bf2e312e347\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") "
Jul 01 21:29:37.010170 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.010110    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ldmk\" (UniqueName: \"kubernetes.io/projected/9695d6e5-2cba-462f-9194-0bf2e312e347-kube-api-access-2ldmk\") pod \"9695d6e5-2cba-462f-9194-0bf2e312e347\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") "
Jul 01 21:29:37.010170 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.010141    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-tmp-dir\") pod \"9695d6e5-2cba-462f-9194-0bf2e312e347\" (UID: \"9695d6e5-2cba-462f-9194-0bf2e312e347\") "
Jul 01 21:29:37.010327 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.010188    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-model-cache" (OuterVolumeSpecName: "model-cache") pod "9695d6e5-2cba-462f-9194-0bf2e312e347" (UID: "9695d6e5-2cba-462f-9194-0bf2e312e347"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.010649 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.010378    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.010649 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.010601    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-home" (OuterVolumeSpecName: "home") pod "9695d6e5-2cba-462f-9194-0bf2e312e347" (UID: "9695d6e5-2cba-462f-9194-0bf2e312e347"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.013706 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.013672    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9695d6e5-2cba-462f-9194-0bf2e312e347-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9695d6e5-2cba-462f-9194-0bf2e312e347" (UID: "9695d6e5-2cba-462f-9194-0bf2e312e347"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:29:37.013706 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.013695    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9695d6e5-2cba-462f-9194-0bf2e312e347-kube-api-access-2ldmk" (OuterVolumeSpecName: "kube-api-access-2ldmk") pod "9695d6e5-2cba-462f-9194-0bf2e312e347" (UID: "9695d6e5-2cba-462f-9194-0bf2e312e347"). InnerVolumeSpecName "kube-api-access-2ldmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:29:37.013849 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.013837    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-dshm" (OuterVolumeSpecName: "dshm") pod "9695d6e5-2cba-462f-9194-0bf2e312e347" (UID: "9695d6e5-2cba-462f-9194-0bf2e312e347"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.026711 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.026680    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9695d6e5-2cba-462f-9194-0bf2e312e347" (UID: "9695d6e5-2cba-462f-9194-0bf2e312e347"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.078697 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.078669    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:29:37.085185 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.085160    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9695d6e5-2cba-462f-9194-0bf2e312e347" (UID: "9695d6e5-2cba-462f-9194-0bf2e312e347"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.110998 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.110975    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldmk\" (UniqueName: \"kubernetes.io/projected/9695d6e5-2cba-462f-9194-0bf2e312e347-kube-api-access-2ldmk\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.110998 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.110999    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.111145 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.111009    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.111145 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.111017    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9695d6e5-2cba-462f-9194-0bf2e312e347-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.111145 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.111025    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.111145 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.111033    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9695d6e5-2cba-462f-9194-0bf2e312e347-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.212038 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.212013    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brp4h\" (UniqueName: \"kubernetes.io/projected/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kube-api-access-brp4h\") pod \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") "
Jul 01 21:29:37.212171 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.212059    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kserve-provision-location\") pod \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") "
Jul 01 21:29:37.212171 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.212100    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-dshm\") pod \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") "
Jul 01 21:29:37.212171 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.212117    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tmp-dir\") pod \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") "
Jul 01 21:29:37.212171 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.212139    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-home\") pod \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") "
Jul 01 21:29:37.212379 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.212181    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-model-cache\") pod \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") "
Jul 01 21:29:37.212379 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.212205    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tls-certs\") pod \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\" (UID: \"f27e8da6-a33a-4a8c-bf54-739f9221f4f6\") "
Jul 01 21:29:37.213480 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.213427    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-home" (OuterVolumeSpecName: "home") pod "f27e8da6-a33a-4a8c-bf54-739f9221f4f6" (UID: "f27e8da6-a33a-4a8c-bf54-739f9221f4f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.213720 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.213691    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f27e8da6-a33a-4a8c-bf54-739f9221f4f6" (UID: "f27e8da6-a33a-4a8c-bf54-739f9221f4f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.213920 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.213901    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.213986 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.213928    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.215139 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.215104    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f27e8da6-a33a-4a8c-bf54-739f9221f4f6" (UID: "f27e8da6-a33a-4a8c-bf54-739f9221f4f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:29:37.224166 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.223979    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kube-api-access-brp4h" (OuterVolumeSpecName: "kube-api-access-brp4h") pod "f27e8da6-a33a-4a8c-bf54-739f9221f4f6" (UID: "f27e8da6-a33a-4a8c-bf54-739f9221f4f6"). InnerVolumeSpecName "kube-api-access-brp4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:29:37.225624 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.225595    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-dshm" (OuterVolumeSpecName: "dshm") pod "f27e8da6-a33a-4a8c-bf54-739f9221f4f6" (UID: "f27e8da6-a33a-4a8c-bf54-739f9221f4f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.229947 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.229897    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f27e8da6-a33a-4a8c-bf54-739f9221f4f6" (UID: "f27e8da6-a33a-4a8c-bf54-739f9221f4f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.281986 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.281912    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f27e8da6-a33a-4a8c-bf54-739f9221f4f6" (UID: "f27e8da6-a33a-4a8c-bf54-739f9221f4f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:29:37.314540 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.314514    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.314540 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.314537    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brp4h\" (UniqueName: \"kubernetes.io/projected/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kube-api-access-brp4h\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.314689 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.314547    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.314689 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.314555    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.314689 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.314563    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f27e8da6-a33a-4a8c-bf54-739f9221f4f6-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:29:37.369273 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.369237    2573 generic.go:356] "Generic (PLEG): container finished" podID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerID="f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722" exitCode=0
Jul 01 21:29:37.369443 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.369315    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"
Jul 01 21:29:37.369443 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.369310    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" event={"ID":"9695d6e5-2cba-462f-9194-0bf2e312e347","Type":"ContainerDied","Data":"f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722"}
Jul 01 21:29:37.369443 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.369397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg" event={"ID":"9695d6e5-2cba-462f-9194-0bf2e312e347","Type":"ContainerDied","Data":"1cd9f694fb87ca7a878188a813e5787f2e885607d8e6e188a1583f0c34f6846c"}
Jul 01 21:29:37.369443 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.369428    2573 scope.go:117] "RemoveContainer" containerID="f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722"
Jul 01 21:29:37.371488 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.371465    2573 generic.go:356] "Generic (PLEG): container finished" podID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerID="efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3" exitCode=0
Jul 01 21:29:37.371488 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.371484    2573 generic.go:356] "Generic (PLEG): container finished" podID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerID="3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054" exitCode=0
Jul 01 21:29:37.371688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.371519    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" event={"ID":"f27e8da6-a33a-4a8c-bf54-739f9221f4f6","Type":"ContainerDied","Data":"efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3"}
Jul 01 21:29:37.371688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.371605    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" event={"ID":"f27e8da6-a33a-4a8c-bf54-739f9221f4f6","Type":"ContainerDied","Data":"3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054"}
Jul 01 21:29:37.371688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.371549    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"
Jul 01 21:29:37.371688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.371641    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp" event={"ID":"f27e8da6-a33a-4a8c-bf54-739f9221f4f6","Type":"ContainerDied","Data":"9b63a0cc731d32506587fa2e0215f6a3a161765278d6c8b9b401db234ccc4437"}
Jul 01 21:29:37.380120 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.380062    2573 scope.go:117] "RemoveContainer" containerID="cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3"
Jul 01 21:29:37.388590 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.388566    2573 scope.go:117] "RemoveContainer" containerID="f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722"
Jul 01 21:29:37.388840 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:29:37.388818    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722\": container with ID starting with f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722 not found: ID does not exist" containerID="f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722"
Jul 01 21:29:37.388984 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.388846    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722"} err="failed to get container status \"f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722\": rpc error: code = NotFound desc = could not find container \"f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722\": container with ID starting with f7123f57ea1f0ecd65f4f37d28b7c94133671bc72ad37d0352901d3ed7b0e722 not found: ID does not exist"
Jul 01 21:29:37.388984 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.388880    2573 scope.go:117] "RemoveContainer" containerID="cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3"
Jul 01 21:29:37.389496 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:29:37.389480    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3\": container with ID starting with cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3 not found: ID does not exist" containerID="cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3"
Jul 01 21:29:37.389577 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.389497    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3"} err="failed to get container status \"cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3\": rpc error: code = NotFound desc = could not find container \"cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3\": container with ID starting with cba2388c253fcc7307e5d36c9972d26657b5597f42588dfb29b00ef29c0393a3 not found: ID does not exist"
Jul 01 21:29:37.389577 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.389510    2573 scope.go:117] "RemoveContainer" containerID="efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3"
Jul 01 21:29:37.393674 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.393654    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"]
Jul 01 21:29:37.397373 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.397348    2573 scope.go:117] "RemoveContainer" containerID="d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd"
Jul 01 21:29:37.399926 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.399907    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg"]
Jul 01 21:29:37.405435 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.405414    2573 scope.go:117] "RemoveContainer" containerID="3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054"
Jul 01 21:29:37.411069 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.411044    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"]
Jul 01 21:29:37.413049 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.413028    2573 scope.go:117] "RemoveContainer" containerID="efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3"
Jul 01 21:29:37.413335 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:29:37.413310    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3\": container with ID starting with efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3 not found: ID does not exist" containerID="efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3"
Jul 01 21:29:37.413416 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.413342    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3"} err="failed to get container status \"efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3\": rpc error: code = NotFound desc = could not find container \"efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3\": container with ID starting with efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3 not found: ID does not exist"
Jul 01 21:29:37.413416 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.413358    2573 scope.go:117] "RemoveContainer" containerID="d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd"
Jul 01 21:29:37.413623 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:29:37.413602    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd\": container with ID starting with d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd not found: ID does not exist" containerID="d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd"
Jul 01 21:29:37.413679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.413630    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd"} err="failed to get container status \"d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd\": rpc error: code = NotFound desc = could not find container \"d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd\": container with ID starting with d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd not found: ID does not exist"
Jul 01 21:29:37.413679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.413648    2573 scope.go:117] "RemoveContainer" containerID="3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054"
Jul 01 21:29:37.413913 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:29:37.413896    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054\": container with ID starting with 3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054 not found: ID does not exist" containerID="3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054"
Jul 01 21:29:37.413976 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.413916    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054"} err="failed to get container status \"3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054\": rpc error: code = NotFound desc = could not find container \"3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054\": container with ID starting with 3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054 not found: ID does not exist"
Jul 01 21:29:37.413976 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.413929    2573 scope.go:117] "RemoveContainer" containerID="efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3"
Jul 01 21:29:37.414192 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.414174    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3"} err="failed to get container status \"efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3\": rpc error: code = NotFound desc = could not find container \"efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3\": container with ID starting with efa2a98f32fdd3db92d74bceedf623d4a3c5c3478bb737fa920ce8a85db58fd3 not found: ID does not exist"
Jul 01 21:29:37.414257 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.414192    2573 scope.go:117] "RemoveContainer" containerID="d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd"
Jul 01 21:29:37.414421 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.414393    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd"} err="failed to get container status \"d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd\": rpc error: code = NotFound desc = could not find container \"d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd\": container with ID starting with d91b097a4ae55a823ef5d6bb9c31e2261aa8dab75af15c017ea065e69f3f7bbd not found: ID does not exist"
Jul 01 21:29:37.414421 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.414410    2573 scope.go:117] "RemoveContainer" containerID="3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054"
Jul 01 21:29:37.414534 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.414516    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp"]
Jul 01 21:29:37.414599 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.414570    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054"} err="failed to get container status \"3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054\": rpc error: code = NotFound desc = could not find container \"3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054\": container with ID starting with 3377182a08dd06937e11679a525b1e825dd56e56f3f5133f7a8f42f0c7a88054 not found: ID does not exist"
Jul 01 21:29:37.591951 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.591916    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" path="/var/lib/kubelet/pods/9695d6e5-2cba-462f-9194-0bf2e312e347/volumes"
Jul 01 21:29:37.592357 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:37.592342    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" path="/var/lib/kubelet/pods/f27e8da6-a33a-4a8c-bf54-739f9221f4f6/volumes"
Jul 01 21:29:45.505831 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:45.505789    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:29:47.195546 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195511    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"]
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195808    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="storage-initializer"
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195819    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="storage-initializer"
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195829    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="storage-initializer"
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195835    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="storage-initializer"
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195841    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main"
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195847    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main"
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195869    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="llm-d-routing-sidecar"
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195874    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="llm-d-routing-sidecar"
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195890    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main"
Jul 01 21:29:47.195913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195895    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main"
Jul 01 21:29:47.196223 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195947    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9695d6e5-2cba-462f-9194-0bf2e312e347" containerName="main"
Jul 01 21:29:47.196223 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195955    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="main"
Jul 01 21:29:47.196223 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.195963    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27e8da6-a33a-4a8c-bf54-739f9221f4f6" containerName="llm-d-routing-sidecar"
Jul 01 21:29:47.197732 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.197716    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.200974 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.200942    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 01 21:29:47.212661 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.212638    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"]
Jul 01 21:29:47.296220 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.296195    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.296344 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.296234    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.296344 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.296262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.296344 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.296312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.296344 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.296329    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgvrq\" (UniqueName: \"kubernetes.io/projected/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kube-api-access-fgvrq\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.296500 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.296359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.296500 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.296389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397272 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397277    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397523 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397295    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgvrq\" (UniqueName: \"kubernetes.io/projected/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kube-api-access-fgvrq\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397523 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397523 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397523 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397490    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397736 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397736 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397705    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397762    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397826    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.397985 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.397888    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.399707 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.399680    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.399972 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.399953    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.405903 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.405875    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgvrq\" (UniqueName: \"kubernetes.io/projected/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kube-api-access-fgvrq\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.507800 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.507718    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:47.648234 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:47.648207    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"]
Jul 01 21:29:48.423158 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:48.423122    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" event={"ID":"b961d55b-7bd2-47fc-8050-fc64b98b2e9a","Type":"ContainerStarted","Data":"ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b"}
Jul 01 21:29:48.423515 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:48.423174    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" event={"ID":"b961d55b-7bd2-47fc-8050-fc64b98b2e9a","Type":"ContainerStarted","Data":"dffd625a7f9a2e39f4b03b1851397d543d32bc0788ef09a955f2360ab024d148"}
Jul 01 21:29:52.443884 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:52.443834    2573 generic.go:356] "Generic (PLEG): container finished" podID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerID="ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b" exitCode=0
Jul 01 21:29:52.444358 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:52.443891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" event={"ID":"b961d55b-7bd2-47fc-8050-fc64b98b2e9a","Type":"ContainerDied","Data":"ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b"}
Jul 01 21:29:53.451598 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:53.451567    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" event={"ID":"b961d55b-7bd2-47fc-8050-fc64b98b2e9a","Type":"ContainerStarted","Data":"0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820"}
Jul 01 21:29:53.476544 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:53.476484    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podStartSLOduration=2.536314731 podStartE2EDuration="6.476462045s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.940147314s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:29:53.472681938 +0000 UTC m=+2626.384016567" watchObservedRunningTime="2026-07-01 21:29:53.476462045 +0000 UTC m=+2626.387796672"
Jul 01 21:29:53.508236 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:53.508206    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:55.512672 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:55.512628    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:29:57.508825 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:57.508784    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:29:57.510479 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:29:57.510448    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:30:05.505122 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:05.505077    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:30:07.508953 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:07.508906    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:30:15.505195 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:15.505141    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:30:17.508445 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:17.508405    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:30:25.505447 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:25.505390    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:30:27.508963 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:27.508922    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:30:35.505403 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:35.505341    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:30:37.508909 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:37.508870    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:30:45.515491 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:45.515451    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:30:45.523918 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:45.523893    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:30:46.746995 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:46.746961    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"]
Jul 01 21:30:47.508156 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:47.508117    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:30:57.510256 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:30:57.510015    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:31:01.771275 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:01.771171    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" containerID="cri-o://e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521" gracePeriod=45
Jul 01 21:31:02.517096 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.517048    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 01 21:31:02.724875 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.724833    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:31:02.759752 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.759724    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kserve-provision-location\") pod \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") "
Jul 01 21:31:02.759932 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.759770    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tmp-dir\") pod \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") "
Jul 01 21:31:02.759932 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.759791    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-dshm\") pod \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") "
Jul 01 21:31:02.759932 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.759815    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tls-certs\") pod \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") "
Jul 01 21:31:02.759932 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.759836    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-model-cache\") pod \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") "
Jul 01 21:31:02.759932 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.759893    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84z62\" (UniqueName: \"kubernetes.io/projected/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kube-api-access-84z62\") pod \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") "
Jul 01 21:31:02.760178 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.759935    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-home\") pod \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\" (UID: \"8ce1e791-4cc1-4780-8844-7a2f9f4aea15\") "
Jul 01 21:31:02.769917 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.761435    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ce1e791-4cc1-4780-8844-7a2f9f4aea15" (UID: "8ce1e791-4cc1-4780-8844-7a2f9f4aea15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:31:02.769917 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.762017    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-home" (OuterVolumeSpecName: "home") pod "8ce1e791-4cc1-4780-8844-7a2f9f4aea15" (UID: "8ce1e791-4cc1-4780-8844-7a2f9f4aea15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:31:02.770976 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.770929    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-dshm" (OuterVolumeSpecName: "dshm") pod "8ce1e791-4cc1-4780-8844-7a2f9f4aea15" (UID: "8ce1e791-4cc1-4780-8844-7a2f9f4aea15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:31:02.772576 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.771793    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ce1e791-4cc1-4780-8844-7a2f9f4aea15" (UID: "8ce1e791-4cc1-4780-8844-7a2f9f4aea15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:31:02.773910 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.773868    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kube-api-access-84z62" (OuterVolumeSpecName: "kube-api-access-84z62") pod "8ce1e791-4cc1-4780-8844-7a2f9f4aea15" (UID: "8ce1e791-4cc1-4780-8844-7a2f9f4aea15"). InnerVolumeSpecName "kube-api-access-84z62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:31:02.789610 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.789578    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ce1e791-4cc1-4780-8844-7a2f9f4aea15" (UID: "8ce1e791-4cc1-4780-8844-7a2f9f4aea15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:31:02.792108 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.792083    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerID="e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521" exitCode=0
Jul 01 21:31:02.792236 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.792170    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"
Jul 01 21:31:02.792236 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.792166    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" event={"ID":"8ce1e791-4cc1-4780-8844-7a2f9f4aea15","Type":"ContainerDied","Data":"e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521"}
Jul 01 21:31:02.792347 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.792281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh" event={"ID":"8ce1e791-4cc1-4780-8844-7a2f9f4aea15","Type":"ContainerDied","Data":"7770ed1235c67cfee8fcc041018ed205b78cd0b9640a67f187172e804e1c5158"}
Jul 01 21:31:02.792347 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.792305    2573 scope.go:117] "RemoveContainer" containerID="e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521"
Jul 01 21:31:02.802168 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.802147    2573 scope.go:117] "RemoveContainer" containerID="faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb"
Jul 01 21:31:02.811623 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.811604    2573 scope.go:117] "RemoveContainer" containerID="e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521"
Jul 01 21:31:02.811904 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:31:02.811884    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521\": container with ID starting with e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521 not found: ID does not exist" containerID="e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521"
Jul 01 21:31:02.812000 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.811911    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521"} err="failed to get container status \"e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521\": rpc error: code = NotFound desc = could not find container \"e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521\": container with ID starting with e476d038f9244d3a436f2274a66f11a9dfb2a687f187051319e61cae14fb4521 not found: ID does not exist"
Jul 01 21:31:02.812000 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.811929    2573 scope.go:117] "RemoveContainer" containerID="faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb"
Jul 01 21:31:02.812160 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:31:02.812144    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb\": container with ID starting with faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb not found: ID does not exist" containerID="faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb"
Jul 01 21:31:02.812231 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.812162    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb"} err="failed to get container status \"faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb\": rpc error: code = NotFound desc = could not find container \"faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb\": container with ID starting with faec71e0b649a5f6441d6dec6c839b69c52e5c7b43d9c5f50bf6ca89ee6f0afb not found: ID does not exist"
Jul 01 21:31:02.834457 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.834399    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ce1e791-4cc1-4780-8844-7a2f9f4aea15" (UID: "8ce1e791-4cc1-4780-8844-7a2f9f4aea15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:31:02.860804 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.860759    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84z62\" (UniqueName: \"kubernetes.io/projected/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kube-api-access-84z62\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:31:02.860804 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.860787    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:31:02.860804 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.860797    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:31:02.861128 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.860826    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:31:02.861128 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.860835    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:31:02.861128 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.860843    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:31:02.861128 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:02.860850    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce1e791-4cc1-4780-8844-7a2f9f4aea15-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:31:03.114546 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:03.114521    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"]
Jul 01 21:31:03.122388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:03.122363    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh"]
Jul 01 21:31:03.590686 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:03.590649    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" path="/var/lib/kubelet/pods/8ce1e791-4cc1-4780-8844-7a2f9f4aea15/volumes"
Jul 01 21:31:07.508069 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:07.508030    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:31:07.761672 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:07.761585    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:31:07.767321 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:07.767300    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:31:07.769185 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:07.769160    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:31:07.781167 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:07.781148    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:31:17.508780 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:17.508735    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:31:22.798138 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.798108    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"]
Jul 01 21:31:22.798582 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.798549    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main"
Jul 01 21:31:22.798582 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.798567    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main"
Jul 01 21:31:22.798705 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.798604    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="storage-initializer"
Jul 01 21:31:22.798705 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.798614    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="storage-initializer"
Jul 01 21:31:22.798705 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.798700    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce1e791-4cc1-4780-8844-7a2f9f4aea15" containerName="main"
Jul 01 21:31:22.801525 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.801507    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:22.804203 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.804187    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 01 21:31:22.813018 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.812995    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"]
Jul 01 21:31:22.920163 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.920134    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tmp-dir\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:22.920163 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.920167    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tls-certs\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:22.920385 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.920187    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-home\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:22.920385 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.920218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-model-cache\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:22.920385 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.920243    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-dshm\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:22.920385 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.920263    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnqg\" (UniqueName: \"kubernetes.io/projected/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kube-api-access-ltnqg\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:22.920385 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:22.920317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kserve-provision-location\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021156 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021126    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-home\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021333 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021165    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-model-cache\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021333 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-dshm\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021333 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021204    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnqg\" (UniqueName: \"kubernetes.io/projected/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kube-api-access-ltnqg\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021333 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kserve-provision-location\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021564 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021328    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tmp-dir\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021564 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tls-certs\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021703 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021679    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-home\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021868 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021745    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tmp-dir\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.021960 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021759    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kserve-provision-location\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.022025 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.021958    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-model-cache\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.023999 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.023972    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-dshm\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.024202 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.024188    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tls-certs\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.031885 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.031848    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnqg\" (UniqueName: \"kubernetes.io/projected/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kube-api-access-ltnqg\") pod \"stop-feature-test-kserve-8486474f84-dz7zq\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.114823 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.114796    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:23.249270 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.249245    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"]
Jul 01 21:31:23.251734 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:31:23.251704    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c23f8d_1ca0_44e8_a53a_454e9b5fe141.slice/crio-7b91058b56d18f8b203ab41cbe725deb22aeed17f29928eb3fc88a093ed1b6b2 WatchSource:0}: Error finding container 7b91058b56d18f8b203ab41cbe725deb22aeed17f29928eb3fc88a093ed1b6b2: Status 404 returned error can't find the container with id 7b91058b56d18f8b203ab41cbe725deb22aeed17f29928eb3fc88a093ed1b6b2
Jul 01 21:31:23.890989 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.890943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" event={"ID":"84c23f8d-1ca0-44e8-a53a-454e9b5fe141","Type":"ContainerStarted","Data":"86e4c6588d4cd67b3b1f77cd726880b63480a3bee9aeec962106f46b52610426"}
Jul 01 21:31:23.891341 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:23.891015    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" event={"ID":"84c23f8d-1ca0-44e8-a53a-454e9b5fe141","Type":"ContainerStarted","Data":"7b91058b56d18f8b203ab41cbe725deb22aeed17f29928eb3fc88a093ed1b6b2"}
Jul 01 21:31:27.509165 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:27.509124    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:31:27.913198 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:27.913166    2573 generic.go:356] "Generic (PLEG): container finished" podID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerID="86e4c6588d4cd67b3b1f77cd726880b63480a3bee9aeec962106f46b52610426" exitCode=0
Jul 01 21:31:27.913348 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:27.913242    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" event={"ID":"84c23f8d-1ca0-44e8-a53a-454e9b5fe141","Type":"ContainerDied","Data":"86e4c6588d4cd67b3b1f77cd726880b63480a3bee9aeec962106f46b52610426"}
Jul 01 21:31:28.920314 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:28.920269    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" event={"ID":"84c23f8d-1ca0-44e8-a53a-454e9b5fe141","Type":"ContainerStarted","Data":"185f571b26e5b1db8ae7d94d2dd16a47f5ab89cc604c4a34ebaf9df4a694adae"}
Jul 01 21:31:28.944014 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:28.943965    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podStartSLOduration=2.679296107 podStartE2EDuration="6.943949883s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.264653776s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:31:28.941337129 +0000 UTC m=+2721.852671756" watchObservedRunningTime="2026-07-01 21:31:28.943949883 +0000 UTC m=+2721.855284509"
Jul 01 21:31:29.115226 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:29.115198    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:33.115933 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:33.115893    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:31:33.117150 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:33.117124    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:31:37.509113 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:37.509069    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:31:43.115694 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:43.115651    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:31:47.518627 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:47.518593    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:31:47.526867 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:47.526832    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:31:53.115980 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:31:53.115939    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:32:03.115668 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:32:03.115628    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:32:13.116184 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:32:13.116143    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:32:23.115839 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:32:23.115794    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:32:33.115726 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:32:33.115682    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:32:43.115176 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:32:43.115129    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:32:53.116095 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:32:53.116049    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:32:54.913830 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:32:54.913792    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"]
Jul 01 21:33:03.115262 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:03.115215    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:33:09.937511 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:09.937440    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" containerID="cri-o://0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820" gracePeriod=45
Jul 01 21:33:10.519611 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.519566    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 01 21:33:10.900831 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.900804    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:33:10.924936 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.924910    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kserve-provision-location\") pod \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") "
Jul 01 21:33:10.925113 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.924953    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tmp-dir\") pod \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") "
Jul 01 21:33:10.925113 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.924986    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-model-cache\") pod \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") "
Jul 01 21:33:10.925113 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.925021    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgvrq\" (UniqueName: \"kubernetes.io/projected/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kube-api-access-fgvrq\") pod \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") "
Jul 01 21:33:10.925113 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.925063    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tls-certs\") pod \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") "
Jul 01 21:33:10.925113 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.925108    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-dshm\") pod \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") "
Jul 01 21:33:10.925403 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.925143    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-home\") pod \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\" (UID: \"b961d55b-7bd2-47fc-8050-fc64b98b2e9a\") "
Jul 01 21:33:10.925403 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.925305    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b961d55b-7bd2-47fc-8050-fc64b98b2e9a" (UID: "b961d55b-7bd2-47fc-8050-fc64b98b2e9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:10.925521 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.925408    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:10.926985 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.926943    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-home" (OuterVolumeSpecName: "home") pod "b961d55b-7bd2-47fc-8050-fc64b98b2e9a" (UID: "b961d55b-7bd2-47fc-8050-fc64b98b2e9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:10.928126 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.928101    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-dshm" (OuterVolumeSpecName: "dshm") pod "b961d55b-7bd2-47fc-8050-fc64b98b2e9a" (UID: "b961d55b-7bd2-47fc-8050-fc64b98b2e9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:10.928234 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.928198    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kube-api-access-fgvrq" (OuterVolumeSpecName: "kube-api-access-fgvrq") pod "b961d55b-7bd2-47fc-8050-fc64b98b2e9a" (UID: "b961d55b-7bd2-47fc-8050-fc64b98b2e9a"). InnerVolumeSpecName "kube-api-access-fgvrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:33:10.928919 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.928893    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b961d55b-7bd2-47fc-8050-fc64b98b2e9a" (UID: "b961d55b-7bd2-47fc-8050-fc64b98b2e9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:33:10.940332 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.940304    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b961d55b-7bd2-47fc-8050-fc64b98b2e9a" (UID: "b961d55b-7bd2-47fc-8050-fc64b98b2e9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:10.995792 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:10.995733    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b961d55b-7bd2-47fc-8050-fc64b98b2e9a" (UID: "b961d55b-7bd2-47fc-8050-fc64b98b2e9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:11.026333 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.026303    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:11.026333 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.026333    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgvrq\" (UniqueName: \"kubernetes.io/projected/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kube-api-access-fgvrq\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:11.026506 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.026347    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:11.026506 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.026356    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:11.026506 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.026363    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:11.026506 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.026370    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b961d55b-7bd2-47fc-8050-fc64b98b2e9a-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:11.412494 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.412462    2573 generic.go:356] "Generic (PLEG): container finished" podID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerID="0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820" exitCode=0
Jul 01 21:33:11.412660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.412522    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"
Jul 01 21:33:11.412660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.412524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" event={"ID":"b961d55b-7bd2-47fc-8050-fc64b98b2e9a","Type":"ContainerDied","Data":"0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820"}
Jul 01 21:33:11.412660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.412644    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm" event={"ID":"b961d55b-7bd2-47fc-8050-fc64b98b2e9a","Type":"ContainerDied","Data":"dffd625a7f9a2e39f4b03b1851397d543d32bc0788ef09a955f2360ab024d148"}
Jul 01 21:33:11.412769 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.412671    2573 scope.go:117] "RemoveContainer" containerID="0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820"
Jul 01 21:33:11.422304 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.422289    2573 scope.go:117] "RemoveContainer" containerID="ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b"
Jul 01 21:33:11.430446 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.430427    2573 scope.go:117] "RemoveContainer" containerID="0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820"
Jul 01 21:33:11.430697 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:33:11.430679    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820\": container with ID starting with 0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820 not found: ID does not exist" containerID="0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820"
Jul 01 21:33:11.430784 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.430704    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820"} err="failed to get container status \"0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820\": rpc error: code = NotFound desc = could not find container \"0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820\": container with ID starting with 0fb051a8d4d500eb28f786ebf8bf440b2b02d1330e171e0f7e1d536fac372820 not found: ID does not exist"
Jul 01 21:33:11.430784 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.430719    2573 scope.go:117] "RemoveContainer" containerID="ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b"
Jul 01 21:33:11.430948 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:33:11.430929    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b\": container with ID starting with ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b not found: ID does not exist" containerID="ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b"
Jul 01 21:33:11.431009 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.430961    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b"} err="failed to get container status \"ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b\": rpc error: code = NotFound desc = could not find container \"ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b\": container with ID starting with ef21a65b5723bea45f80ffaf38f9a166a0fe257d293bd1eb586226b5e42d928b not found: ID does not exist"
Jul 01 21:33:11.436037 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.436018    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"]
Jul 01 21:33:11.439372 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.439352    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm"]
Jul 01 21:33:11.591497 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:11.591466    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" path="/var/lib/kubelet/pods/b961d55b-7bd2-47fc-8050-fc64b98b2e9a/volumes"
Jul 01 21:33:13.115637 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:13.115592    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:33:23.125067 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:23.125029    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:33:23.132696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:23.132675    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:33:26.803660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.803625    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 01 21:33:26.804106 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.803944    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main"
Jul 01 21:33:26.804106 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.803956    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main"
Jul 01 21:33:26.804106 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.803967    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="storage-initializer"
Jul 01 21:33:26.804106 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.803973    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="storage-initializer"
Jul 01 21:33:26.804106 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.804029    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b961d55b-7bd2-47fc-8050-fc64b98b2e9a" containerName="main"
Jul 01 21:33:26.806817 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.806799    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.809585 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.809563    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 01 21:33:26.809660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.809607    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-4x7qh\""
Jul 01 21:33:26.817982 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.817961    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 01 21:33:26.847778 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.847754    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngbls\" (UniqueName: \"kubernetes.io/projected/77707810-9d32-4bd3-b786-5708753ccc8c-kube-api-access-ngbls\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.847925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.847783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.847925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.847806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.847925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.847870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.848029 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.847955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.848029 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.848008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.848095 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.848034    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77707810-9d32-4bd3-b786-5708753ccc8c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.851023 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.851001    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 01 21:33:26.854122 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.854108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:26.867481 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.867461    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 01 21:33:26.949342 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949310    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbls\" (UniqueName: \"kubernetes.io/projected/77707810-9d32-4bd3-b786-5708753ccc8c-kube-api-access-ngbls\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.949492 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.949492 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.949492 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:26.949492 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949426    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:26.949492 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949454    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.949729 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.949729 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:26.949729 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949586    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24b5\" (UniqueName: \"kubernetes.io/projected/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kube-api-access-z24b5\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:26.949729 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:26.949729 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949678    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:26.949729 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949698    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.949729 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949716    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.950075 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949757    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77707810-9d32-4bd3-b786-5708753ccc8c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.950075 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:26.950075 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949789    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.950075 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949844    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.950075 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.949992    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.951596 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.951580    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.952077 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.952060    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77707810-9d32-4bd3-b786-5708753ccc8c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:26.956912 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:26.956892    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbls\" (UniqueName: \"kubernetes.io/projected/77707810-9d32-4bd3-b786-5708753ccc8c-kube-api-access-ngbls\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:27.050254 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050374 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050288    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050374 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050374 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050548 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24b5\" (UniqueName: \"kubernetes.io/projected/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kube-api-access-z24b5\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050548 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050548 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050450    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050548 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050518    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050768 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050573    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050768 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050611    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.050768 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.050707    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.052490 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.052461    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.052608 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.052589    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.058292 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.058246    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24b5\" (UniqueName: \"kubernetes.io/projected/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kube-api-access-z24b5\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.117295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.117265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:27.163461 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.163437    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:33:27.262553 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.262523    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 01 21:33:27.265121 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:33:27.265089    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77707810_9d32_4bd3_b786_5708753ccc8c.slice/crio-7110694c8f051e9d0067ffefa148b6db845217eb6d031c79cf8689b8782460e9 WatchSource:0}: Error finding container 7110694c8f051e9d0067ffefa148b6db845217eb6d031c79cf8689b8782460e9: Status 404 returned error can't find the container with id 7110694c8f051e9d0067ffefa148b6db845217eb6d031c79cf8689b8782460e9
Jul 01 21:33:27.308790 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.308718    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 01 21:33:27.311085 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:33:27.311063    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3101b4e4_2fd9_4ce0_a351_1deca1cd1bc2.slice/crio-f0f0bd57a7a73c9779604481b312fe0a559751a6e2fa524a920de87e8d012542 WatchSource:0}: Error finding container f0f0bd57a7a73c9779604481b312fe0a559751a6e2fa524a920de87e8d012542: Status 404 returned error can't find the container with id f0f0bd57a7a73c9779604481b312fe0a559751a6e2fa524a920de87e8d012542
Jul 01 21:33:27.489251 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.489211    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2","Type":"ContainerStarted","Data":"12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c"}
Jul 01 21:33:27.489452 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.489279    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2","Type":"ContainerStarted","Data":"f0f0bd57a7a73c9779604481b312fe0a559751a6e2fa524a920de87e8d012542"}
Jul 01 21:33:27.490962 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.490932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"77707810-9d32-4bd3-b786-5708753ccc8c","Type":"ContainerStarted","Data":"386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0"}
Jul 01 21:33:27.491077 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:27.490980    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"77707810-9d32-4bd3-b786-5708753ccc8c","Type":"ContainerStarted","Data":"7110694c8f051e9d0067ffefa148b6db845217eb6d031c79cf8689b8782460e9"}
Jul 01 21:33:31.516112 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:31.516080    2573 generic.go:356] "Generic (PLEG): container finished" podID="3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" containerID="12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c" exitCode=0
Jul 01 21:33:31.516471 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:31.516128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2","Type":"ContainerDied","Data":"12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c"}
Jul 01 21:33:32.524367 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:32.524332    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2","Type":"ContainerStarted","Data":"5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221"}
Jul 01 21:33:32.526100 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:32.526074    2573 generic.go:356] "Generic (PLEG): container finished" podID="77707810-9d32-4bd3-b786-5708753ccc8c" containerID="386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0" exitCode=0
Jul 01 21:33:32.526230 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:32.526156    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"77707810-9d32-4bd3-b786-5708753ccc8c","Type":"ContainerDied","Data":"386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0"}
Jul 01 21:33:32.545400 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:32.545354    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.511031814 podStartE2EDuration="6.545333065s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.034301251s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:33:32.544971562 +0000 UTC m=+2845.456306190" watchObservedRunningTime="2026-07-01 21:33:32.545333065 +0000 UTC m=+2845.456667692"
Jul 01 21:33:33.533256 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:33.533216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"77707810-9d32-4bd3-b786-5708753ccc8c","Type":"ContainerStarted","Data":"8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e"}
Jul 01 21:33:33.556419 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:33.556369    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.316777473 podStartE2EDuration="7.556349718s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.239572245s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:33:33.553867791 +0000 UTC m=+2846.465202414" watchObservedRunningTime="2026-07-01 21:33:33.556349718 +0000 UTC m=+2846.467684377"
Jul 01 21:33:34.117909 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:34.117850    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:35.904927 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:35.904892    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"]
Jul 01 21:33:37.117666 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:37.117635    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:33:37.119239 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:37.119208    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:33:47.118630 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:47.118583    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:33:50.928672 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:50.928628    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" containerID="cri-o://185f571b26e5b1db8ae7d94d2dd16a47f5ab89cc604c4a34ebaf9df4a694adae" gracePeriod=45
Jul 01 21:33:51.126070 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.126029    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 01 21:33:51.627396 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.627367    2573 generic.go:356] "Generic (PLEG): container finished" podID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerID="185f571b26e5b1db8ae7d94d2dd16a47f5ab89cc604c4a34ebaf9df4a694adae" exitCode=0
Jul 01 21:33:51.627575 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.627469    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" event={"ID":"84c23f8d-1ca0-44e8-a53a-454e9b5fe141","Type":"ContainerDied","Data":"185f571b26e5b1db8ae7d94d2dd16a47f5ab89cc604c4a34ebaf9df4a694adae"}
Jul 01 21:33:51.681710 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.681683    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:33:51.785191 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.785105    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tmp-dir\") pod \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") "
Jul 01 21:33:51.785191 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.785136    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kserve-provision-location\") pod \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") "
Jul 01 21:33:51.785191 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.785164    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-home\") pod \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") "
Jul 01 21:33:51.785191 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.785188    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltnqg\" (UniqueName: \"kubernetes.io/projected/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kube-api-access-ltnqg\") pod \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") "
Jul 01 21:33:51.785526 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.785231    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-model-cache\") pod \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") "
Jul 01 21:33:51.785526 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.785325    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-dshm\") pod \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") "
Jul 01 21:33:51.785526 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.785353    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tls-certs\") pod \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\" (UID: \"84c23f8d-1ca0-44e8-a53a-454e9b5fe141\") "
Jul 01 21:33:51.786063 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.785899    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-home" (OuterVolumeSpecName: "home") pod "84c23f8d-1ca0-44e8-a53a-454e9b5fe141" (UID: "84c23f8d-1ca0-44e8-a53a-454e9b5fe141"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:51.786295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.786252    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-model-cache" (OuterVolumeSpecName: "model-cache") pod "84c23f8d-1ca0-44e8-a53a-454e9b5fe141" (UID: "84c23f8d-1ca0-44e8-a53a-454e9b5fe141"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:51.787750 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.787727    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84c23f8d-1ca0-44e8-a53a-454e9b5fe141" (UID: "84c23f8d-1ca0-44e8-a53a-454e9b5fe141"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:33:51.788209 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.788187    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kube-api-access-ltnqg" (OuterVolumeSpecName: "kube-api-access-ltnqg") pod "84c23f8d-1ca0-44e8-a53a-454e9b5fe141" (UID: "84c23f8d-1ca0-44e8-a53a-454e9b5fe141"). InnerVolumeSpecName "kube-api-access-ltnqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:33:51.788507 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.788466    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-dshm" (OuterVolumeSpecName: "dshm") pod "84c23f8d-1ca0-44e8-a53a-454e9b5fe141" (UID: "84c23f8d-1ca0-44e8-a53a-454e9b5fe141"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:51.799175 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.799150    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84c23f8d-1ca0-44e8-a53a-454e9b5fe141" (UID: "84c23f8d-1ca0-44e8-a53a-454e9b5fe141"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:51.854946 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.854907    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84c23f8d-1ca0-44e8-a53a-454e9b5fe141" (UID: "84c23f8d-1ca0-44e8-a53a-454e9b5fe141"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:33:51.886084 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.886046    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:51.886084 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.886071    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:51.886084 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.886083    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:51.886084 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.886091    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:51.886437 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.886099    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:51.886437 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.886108    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltnqg\" (UniqueName: \"kubernetes.io/projected/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-kube-api-access-ltnqg\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:51.886437 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:51.886115    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c23f8d-1ca0-44e8-a53a-454e9b5fe141-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:33:52.634178 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:52.634149    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"
Jul 01 21:33:52.634652 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:52.634188    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq" event={"ID":"84c23f8d-1ca0-44e8-a53a-454e9b5fe141","Type":"ContainerDied","Data":"7b91058b56d18f8b203ab41cbe725deb22aeed17f29928eb3fc88a093ed1b6b2"}
Jul 01 21:33:52.634652 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:52.634262    2573 scope.go:117] "RemoveContainer" containerID="185f571b26e5b1db8ae7d94d2dd16a47f5ab89cc604c4a34ebaf9df4a694adae"
Jul 01 21:33:52.645073 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:52.645057    2573 scope.go:117] "RemoveContainer" containerID="86e4c6588d4cd67b3b1f77cd726880b63480a3bee9aeec962106f46b52610426"
Jul 01 21:33:52.658541 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:52.658517    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"]
Jul 01 21:33:52.660038 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:52.660019    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq"]
Jul 01 21:33:53.592917 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:53.592874    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" path="/var/lib/kubelet/pods/84c23f8d-1ca0-44e8-a53a-454e9b5fe141/volumes"
Jul 01 21:33:57.118543 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:57.118499    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:33:59.567210 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.567134    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"]
Jul 01 21:33:59.567567 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.567512    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main"
Jul 01 21:33:59.567567 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.567526    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main"
Jul 01 21:33:59.567567 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.567550    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="storage-initializer"
Jul 01 21:33:59.567567 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.567556    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="storage-initializer"
Jul 01 21:33:59.567718 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.567610    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c23f8d-1ca0-44e8-a53a-454e9b5fe141" containerName="main"
Jul 01 21:33:59.570696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.570677    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.573653 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.573626    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:33:59.574124 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.574098    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 01 21:33:59.593287 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.593259    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"]
Jul 01 21:33:59.653260 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.653220    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-home\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.653422 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.653265    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-model-cache\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.653422 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.653336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjtbc\" (UniqueName: \"kubernetes.io/projected/0ade30b2-6fc9-41f0-b371-f763ded8468e-kube-api-access-gjtbc\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.653557 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.653437    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ade30b2-6fc9-41f0-b371-f763ded8468e-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.653557 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.653479    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ade30b2-6fc9-41f0-b371-f763ded8468e-tls-certs\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.653557 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.653505    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.653557 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.653538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-tmp-dir\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.653776 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.653594    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-dshm\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.754453 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.754409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-home\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.754453 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.754446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-model-cache\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.754713 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.754485    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjtbc\" (UniqueName: \"kubernetes.io/projected/0ade30b2-6fc9-41f0-b371-f763ded8468e-kube-api-access-gjtbc\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.754713 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.754551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ade30b2-6fc9-41f0-b371-f763ded8468e-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.754713 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.754588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ade30b2-6fc9-41f0-b371-f763ded8468e-tls-certs\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.754713 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.754613    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.754713 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.754643    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-tmp-dir\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.755045 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.754838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-dshm\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.755045 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.754898    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-model-cache\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.755171 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.755076    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.755230 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.755182    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-tmp-dir\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.755469 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.755447    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-home\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.755556 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.755477    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ade30b2-6fc9-41f0-b371-f763ded8468e-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.757261 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.757237    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-dshm\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.757480 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.757458    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ade30b2-6fc9-41f0-b371-f763ded8468e-tls-certs\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.762940 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.762916    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjtbc\" (UniqueName: \"kubernetes.io/projected/0ade30b2-6fc9-41f0-b371-f763ded8468e-kube-api-access-gjtbc\") pod \"lora-single-adapter-test-kserve-6b64c6f8b9-2lb72\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:33:59.882166 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:33:59.882063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:34:00.024970 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:00.024803    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"]
Jul 01 21:34:00.028391 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:34:00.028359    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ade30b2_6fc9_41f0_b371_f763ded8468e.slice/crio-6506441905a79058284372783e498acfe6a8fbfb49e24975ae2f31dc8fc05be7 WatchSource:0}: Error finding container 6506441905a79058284372783e498acfe6a8fbfb49e24975ae2f31dc8fc05be7: Status 404 returned error can't find the container with id 6506441905a79058284372783e498acfe6a8fbfb49e24975ae2f31dc8fc05be7
Jul 01 21:34:00.678433 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:00.678396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" event={"ID":"0ade30b2-6fc9-41f0-b371-f763ded8468e","Type":"ContainerStarted","Data":"b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c"}
Jul 01 21:34:00.678801 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:00.678454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" event={"ID":"0ade30b2-6fc9-41f0-b371-f763ded8468e","Type":"ContainerStarted","Data":"6506441905a79058284372783e498acfe6a8fbfb49e24975ae2f31dc8fc05be7"}
Jul 01 21:34:04.702638 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:04.702595    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerID="b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c" exitCode=0
Jul 01 21:34:04.703131 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:04.702665    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" event={"ID":"0ade30b2-6fc9-41f0-b371-f763ded8468e","Type":"ContainerDied","Data":"b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c"}
Jul 01 21:34:05.710535 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:05.710494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" event={"ID":"0ade30b2-6fc9-41f0-b371-f763ded8468e","Type":"ContainerStarted","Data":"9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2"}
Jul 01 21:34:05.733260 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:05.733198    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podStartSLOduration=2.709459262 podStartE2EDuration="6.733178712s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.02371945s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:34:05.731260621 +0000 UTC m=+2878.642595248" watchObservedRunningTime="2026-07-01 21:34:05.733178712 +0000 UTC m=+2878.644513340"
Jul 01 21:34:05.882821 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:05.882770    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:34:07.119384 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:07.119316    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:34:09.883121 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:09.883073    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:34:09.884799 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:09.884771    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 01 21:34:17.118116 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:17.118064    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:34:19.883171 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:19.883121    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 01 21:34:27.117827 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:27.117781    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:34:29.883436 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:29.883396    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 01 21:34:37.117943 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:37.117897    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:34:39.883015 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:39.882970    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 01 21:34:47.118760 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:47.118702    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:34:49.882924 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:49.882887    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 01 21:34:57.118071 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:57.118021    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:34:59.883530 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:34:59.883485    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 01 21:35:07.117639 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:07.117596    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:35:09.882965 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:09.882908    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 01 21:35:17.118090 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:17.118049    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:35:19.883160 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:19.883106    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 01 21:35:27.117967 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:27.117912    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:35:29.893308 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:29.893228    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:35:29.900913 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:29.900886    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:35:37.118645 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:37.118595    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 01 21:35:47.127444 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:47.127415    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:35:47.134960 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:35:47.134935    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:36:07.797172 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:36:07.797140    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:36:07.802984 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:36:07.802961    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:36:07.803715 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:36:07.803697    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:36:07.809331 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:36:07.809315    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:41:07.827113 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:41:07.827084    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:41:07.835014 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:41:07.834995    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:41:07.835151 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:41:07.835085    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:41:07.841876 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:41:07.841846    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:46:07.868031 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:07.867924    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:46:07.874920 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:07.874892    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:46:07.878963 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:07.878935    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:46:07.882584 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:07.882566    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:46:14.611283 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:14.611248    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"]
Jul 01 21:46:20.441847 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:20.441816    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 01 21:46:29.634210 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:29.634174    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" containerID="cri-o://9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2" gracePeriod=45
Jul 01 21:46:29.894612 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:29.894523    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 01 21:46:30.307730 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.307703    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:46:30.326592 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326566    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ade30b2-6fc9-41f0-b371-f763ded8468e-cabundle-cert\") pod \"0ade30b2-6fc9-41f0-b371-f763ded8468e\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") "
Jul 01 21:46:30.326688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326601    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-home\") pod \"0ade30b2-6fc9-41f0-b371-f763ded8468e\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") "
Jul 01 21:46:30.326688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326621    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-model-cache\") pod \"0ade30b2-6fc9-41f0-b371-f763ded8468e\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") "
Jul 01 21:46:30.326688 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326646    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-tmp-dir\") pod \"0ade30b2-6fc9-41f0-b371-f763ded8468e\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") "
Jul 01 21:46:30.326875 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326688    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-kserve-provision-location\") pod \"0ade30b2-6fc9-41f0-b371-f763ded8468e\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") "
Jul 01 21:46:30.326875 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326723    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-dshm\") pod \"0ade30b2-6fc9-41f0-b371-f763ded8468e\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") "
Jul 01 21:46:30.326875 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326776    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ade30b2-6fc9-41f0-b371-f763ded8468e-tls-certs\") pod \"0ade30b2-6fc9-41f0-b371-f763ded8468e\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") "
Jul 01 21:46:30.326875 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326809    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjtbc\" (UniqueName: \"kubernetes.io/projected/0ade30b2-6fc9-41f0-b371-f763ded8468e-kube-api-access-gjtbc\") pod \"0ade30b2-6fc9-41f0-b371-f763ded8468e\" (UID: \"0ade30b2-6fc9-41f0-b371-f763ded8468e\") "
Jul 01 21:46:30.327093 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326916    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ade30b2-6fc9-41f0-b371-f763ded8468e" (UID: "0ade30b2-6fc9-41f0-b371-f763ded8468e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:30.327093 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.326985    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-home" (OuterVolumeSpecName: "home") pod "0ade30b2-6fc9-41f0-b371-f763ded8468e" (UID: "0ade30b2-6fc9-41f0-b371-f763ded8468e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:30.327093 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.327031    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ade30b2-6fc9-41f0-b371-f763ded8468e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0ade30b2-6fc9-41f0-b371-f763ded8468e" (UID: "0ade30b2-6fc9-41f0-b371-f763ded8468e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:46:30.327251 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.327109    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:30.327251 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.327129    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:30.327251 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.327144    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ade30b2-6fc9-41f0-b371-f763ded8468e-cabundle-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:30.329087 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.329059    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ade30b2-6fc9-41f0-b371-f763ded8468e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ade30b2-6fc9-41f0-b371-f763ded8468e" (UID: "0ade30b2-6fc9-41f0-b371-f763ded8468e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:46:30.329304 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.329279    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ade30b2-6fc9-41f0-b371-f763ded8468e-kube-api-access-gjtbc" (OuterVolumeSpecName: "kube-api-access-gjtbc") pod "0ade30b2-6fc9-41f0-b371-f763ded8468e" (UID: "0ade30b2-6fc9-41f0-b371-f763ded8468e"). InnerVolumeSpecName "kube-api-access-gjtbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:46:30.329417 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.329293    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-dshm" (OuterVolumeSpecName: "dshm") pod "0ade30b2-6fc9-41f0-b371-f763ded8468e" (UID: "0ade30b2-6fc9-41f0-b371-f763ded8468e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:30.340635 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.340612    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ade30b2-6fc9-41f0-b371-f763ded8468e" (UID: "0ade30b2-6fc9-41f0-b371-f763ded8468e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:30.394618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.394586    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ade30b2-6fc9-41f0-b371-f763ded8468e" (UID: "0ade30b2-6fc9-41f0-b371-f763ded8468e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:30.428183 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.428124    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:30.428183 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.428145    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:30.428183 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.428153    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ade30b2-6fc9-41f0-b371-f763ded8468e-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:30.428183 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.428162    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ade30b2-6fc9-41f0-b371-f763ded8468e-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:30.428183 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:30.428170    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjtbc\" (UniqueName: \"kubernetes.io/projected/0ade30b2-6fc9-41f0-b371-f763ded8468e-kube-api-access-gjtbc\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:31.145295 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.145260    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerID="9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2" exitCode=0
Jul 01 21:46:31.145783 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.145327    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"
Jul 01 21:46:31.145783 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.145326    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" event={"ID":"0ade30b2-6fc9-41f0-b371-f763ded8468e","Type":"ContainerDied","Data":"9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2"}
Jul 01 21:46:31.145783 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.145452    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72" event={"ID":"0ade30b2-6fc9-41f0-b371-f763ded8468e","Type":"ContainerDied","Data":"6506441905a79058284372783e498acfe6a8fbfb49e24975ae2f31dc8fc05be7"}
Jul 01 21:46:31.145783 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.145477    2573 scope.go:117] "RemoveContainer" containerID="9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2"
Jul 01 21:46:31.155445 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.155425    2573 scope.go:117] "RemoveContainer" containerID="b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c"
Jul 01 21:46:31.163555 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.163535    2573 scope.go:117] "RemoveContainer" containerID="9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2"
Jul 01 21:46:31.163796 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:46:31.163778    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2\": container with ID starting with 9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2 not found: ID does not exist" containerID="9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2"
Jul 01 21:46:31.163840 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.163804    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2"} err="failed to get container status \"9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2\": rpc error: code = NotFound desc = could not find container \"9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2\": container with ID starting with 9fcde5ecdb3c6d6cdca3b99dad3eaadf80efe798b99f5b740aeca7009787fbd2 not found: ID does not exist"
Jul 01 21:46:31.163840 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.163820    2573 scope.go:117] "RemoveContainer" containerID="b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c"
Jul 01 21:46:31.164059 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:46:31.164044    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c\": container with ID starting with b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c not found: ID does not exist" containerID="b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c"
Jul 01 21:46:31.164111 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.164062    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c"} err="failed to get container status \"b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c\": rpc error: code = NotFound desc = could not find container \"b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c\": container with ID starting with b11c61b6fc8934b65006c07b66267c7044bfe9e58dd6812b67aea8d6e10f1b8c not found: ID does not exist"
Jul 01 21:46:31.169147 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.169122    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"]
Jul 01 21:46:31.172466 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.172443    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72"]
Jul 01 21:46:31.590782 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:31.590752    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" path="/var/lib/kubelet/pods/0ade30b2-6fc9-41f0-b371-f763ded8468e/volumes"
Jul 01 21:46:35.472308 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:35.472228    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" containerID="cri-o://8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e" gracePeriod=45
Jul 01 21:46:36.157838 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.157815    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:46:36.166283 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.166262    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-dshm\") pod \"77707810-9d32-4bd3-b786-5708753ccc8c\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") "
Jul 01 21:46:36.166409 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.166302    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77707810-9d32-4bd3-b786-5708753ccc8c-tls-certs\") pod \"77707810-9d32-4bd3-b786-5708753ccc8c\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") "
Jul 01 21:46:36.166409 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.166351    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-kserve-provision-location\") pod \"77707810-9d32-4bd3-b786-5708753ccc8c\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") "
Jul 01 21:46:36.166409 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.166394    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-home\") pod \"77707810-9d32-4bd3-b786-5708753ccc8c\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") "
Jul 01 21:46:36.166583 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.166569    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngbls\" (UniqueName: \"kubernetes.io/projected/77707810-9d32-4bd3-b786-5708753ccc8c-kube-api-access-ngbls\") pod \"77707810-9d32-4bd3-b786-5708753ccc8c\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") "
Jul 01 21:46:36.166660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.166636    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-model-cache\") pod \"77707810-9d32-4bd3-b786-5708753ccc8c\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") "
Jul 01 21:46:36.166721 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.166705    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-tmp-dir\") pod \"77707810-9d32-4bd3-b786-5708753ccc8c\" (UID: \"77707810-9d32-4bd3-b786-5708753ccc8c\") "
Jul 01 21:46:36.167087 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.167050    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "77707810-9d32-4bd3-b786-5708753ccc8c" (UID: "77707810-9d32-4bd3-b786-5708753ccc8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:36.167244 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.167137    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-home" (OuterVolumeSpecName: "home") pod "77707810-9d32-4bd3-b786-5708753ccc8c" (UID: "77707810-9d32-4bd3-b786-5708753ccc8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:36.169087 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.169040    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77707810-9d32-4bd3-b786-5708753ccc8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77707810-9d32-4bd3-b786-5708753ccc8c" (UID: "77707810-9d32-4bd3-b786-5708753ccc8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:46:36.169087 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.169040    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-dshm" (OuterVolumeSpecName: "dshm") pod "77707810-9d32-4bd3-b786-5708753ccc8c" (UID: "77707810-9d32-4bd3-b786-5708753ccc8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:36.169087 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.169042    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77707810-9d32-4bd3-b786-5708753ccc8c-kube-api-access-ngbls" (OuterVolumeSpecName: "kube-api-access-ngbls") pod "77707810-9d32-4bd3-b786-5708753ccc8c" (UID: "77707810-9d32-4bd3-b786-5708753ccc8c"). InnerVolumeSpecName "kube-api-access-ngbls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:46:36.170214 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.170192    2573 generic.go:356] "Generic (PLEG): container finished" podID="77707810-9d32-4bd3-b786-5708753ccc8c" containerID="8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e" exitCode=0
Jul 01 21:46:36.170305 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.170231    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 01 21:46:36.170305 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.170273    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"77707810-9d32-4bd3-b786-5708753ccc8c","Type":"ContainerDied","Data":"8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e"}
Jul 01 21:46:36.170377 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.170332    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"77707810-9d32-4bd3-b786-5708753ccc8c","Type":"ContainerDied","Data":"7110694c8f051e9d0067ffefa148b6db845217eb6d031c79cf8689b8782460e9"}
Jul 01 21:46:36.170377 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.170360    2573 scope.go:117] "RemoveContainer" containerID="8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e"
Jul 01 21:46:36.181140 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.181106    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77707810-9d32-4bd3-b786-5708753ccc8c" (UID: "77707810-9d32-4bd3-b786-5708753ccc8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:36.189242 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.189223    2573 scope.go:117] "RemoveContainer" containerID="386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0"
Jul 01 21:46:36.198617 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.198585    2573 scope.go:117] "RemoveContainer" containerID="8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e"
Jul 01 21:46:36.198921 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:46:36.198898    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e\": container with ID starting with 8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e not found: ID does not exist" containerID="8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e"
Jul 01 21:46:36.198991 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.198934    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e"} err="failed to get container status \"8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e\": rpc error: code = NotFound desc = could not find container \"8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e\": container with ID starting with 8ae7435b600c8c0b4f61f86a680d9440a7348775d788c84c50df21fcd029814e not found: ID does not exist"
Jul 01 21:46:36.198991 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.198958    2573 scope.go:117] "RemoveContainer" containerID="386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0"
Jul 01 21:46:36.199221 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:46:36.199201    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0\": container with ID starting with 386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0 not found: ID does not exist" containerID="386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0"
Jul 01 21:46:36.199283 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.199231    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0"} err="failed to get container status \"386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0\": rpc error: code = NotFound desc = could not find container \"386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0\": container with ID starting with 386121bc2a0b2c63ef4e05b034a3e07ecde11767de4f6067898e4499afe3d2a0 not found: ID does not exist"
Jul 01 21:46:36.240028 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.239962    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77707810-9d32-4bd3-b786-5708753ccc8c" (UID: "77707810-9d32-4bd3-b786-5708753ccc8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:36.267445 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.267423    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:36.267445 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.267444    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77707810-9d32-4bd3-b786-5708753ccc8c-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:36.267558 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.267455    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:36.267558 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.267464    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:36.267558 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.267472    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngbls\" (UniqueName: \"kubernetes.io/projected/77707810-9d32-4bd3-b786-5708753ccc8c-kube-api-access-ngbls\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:36.267558 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.267480    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:36.267558 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.267489    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77707810-9d32-4bd3-b786-5708753ccc8c-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:36.493543 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.493483    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 01 21:46:36.499450 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.499428    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 01 21:46:36.516005 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:36.515979    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 01 21:46:37.128154 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:37.128114    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 01 21:46:37.591305 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:37.591275    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" path="/var/lib/kubelet/pods/77707810-9d32-4bd3-b786-5708753ccc8c/volumes"
Jul 01 21:46:44.658284 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658253    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"]
Jul 01 21:46:44.658619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658542    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main"
Jul 01 21:46:44.658619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658552    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main"
Jul 01 21:46:44.658619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658566    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main"
Jul 01 21:46:44.658619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658571    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main"
Jul 01 21:46:44.658619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658586    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="storage-initializer"
Jul 01 21:46:44.658619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658591    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="storage-initializer"
Jul 01 21:46:44.658619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658601    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="storage-initializer"
Jul 01 21:46:44.658619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658607    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="storage-initializer"
Jul 01 21:46:44.658906 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658675    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="77707810-9d32-4bd3-b786-5708753ccc8c" containerName="main"
Jul 01 21:46:44.658906 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.658683    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ade30b2-6fc9-41f0-b371-f763ded8468e" containerName="main"
Jul 01 21:46:44.663449 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.663432    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.666234 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.666207    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 21:46:44.666356 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.666237    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 01 21:46:44.672558 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.672536    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"]
Jul 01 21:46:44.728106 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.728079    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-home\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.728203 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.728115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-model-cache\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.728203 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.728141    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8274n\" (UniqueName: \"kubernetes.io/projected/a994a194-21f7-4baf-a341-bf881923c36e-kube-api-access-8274n\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.728203 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.728163    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.728203 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.728192    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-dshm\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.728345 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.728233    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.728345 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.728249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a994a194-21f7-4baf-a341-bf881923c36e-tls-certs\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.728345 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.728265    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a994a194-21f7-4baf-a341-bf881923c36e-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829381 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-model-cache\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829478 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8274n\" (UniqueName: \"kubernetes.io/projected/a994a194-21f7-4baf-a341-bf881923c36e-kube-api-access-8274n\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829478 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829478 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-dshm\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829478 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829671 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829484    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a994a194-21f7-4baf-a341-bf881923c36e-tls-certs\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829671 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829498    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a994a194-21f7-4baf-a341-bf881923c36e-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829671 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-home\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829822 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829794    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-model-cache\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829911 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829839    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-home\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.829911 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829890    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.830023 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.829992    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.830215 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.830194    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a994a194-21f7-4baf-a341-bf881923c36e-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.831734 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.831715    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-dshm\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.831875 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.831841    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a994a194-21f7-4baf-a341-bf881923c36e-tls-certs\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.838682 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.838661    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8274n\" (UniqueName: \"kubernetes.io/projected/a994a194-21f7-4baf-a341-bf881923c36e-kube-api-access-8274n\") pod \"lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:44.974726 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:44.974653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:46:45.135043 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:45.135012    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"]
Jul 01 21:46:45.136691 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:46:45.136664    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda994a194_21f7_4baf_a341_bf881923c36e.slice/crio-a2aa0d035c59f1d5e9729124cea9e69f04df142983decf4bf3ea1f59daeea611 WatchSource:0}: Error finding container a2aa0d035c59f1d5e9729124cea9e69f04df142983decf4bf3ea1f59daeea611: Status 404 returned error can't find the container with id a2aa0d035c59f1d5e9729124cea9e69f04df142983decf4bf3ea1f59daeea611
Jul 01 21:46:45.141449 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:45.141425    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:46:45.213375 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:45.213351    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" event={"ID":"a994a194-21f7-4baf-a341-bf881923c36e","Type":"ContainerStarted","Data":"855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88"}
Jul 01 21:46:45.213502 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:45.213397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" event={"ID":"a994a194-21f7-4baf-a341-bf881923c36e","Type":"ContainerStarted","Data":"a2aa0d035c59f1d5e9729124cea9e69f04df142983decf4bf3ea1f59daeea611"}
Jul 01 21:46:51.538792 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:51.538745    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" containerName="main" containerID="cri-o://5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221" gracePeriod=45
Jul 01 21:46:52.198760 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.198737    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:46:52.246130 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.246049    2573 generic.go:356] "Generic (PLEG): container finished" podID="3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" containerID="5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221" exitCode=0
Jul 01 21:46:52.246130 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.246090    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 01 21:46:52.246130 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.246105    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2","Type":"ContainerDied","Data":"5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221"}
Jul 01 21:46:52.246368 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.246174    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2","Type":"ContainerDied","Data":"f0f0bd57a7a73c9779604481b312fe0a559751a6e2fa524a920de87e8d012542"}
Jul 01 21:46:52.246368 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.246206    2573 scope.go:117] "RemoveContainer" containerID="5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221"
Jul 01 21:46:52.254264 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.254249    2573 scope.go:117] "RemoveContainer" containerID="12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c"
Jul 01 21:46:52.262196 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.262181    2573 scope.go:117] "RemoveContainer" containerID="5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221"
Jul 01 21:46:52.262438 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:46:52.262418    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221\": container with ID starting with 5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221 not found: ID does not exist" containerID="5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221"
Jul 01 21:46:52.262485 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.262451    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221"} err="failed to get container status \"5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221\": rpc error: code = NotFound desc = could not find container \"5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221\": container with ID starting with 5a7a683082d3c2d40f87c397a398372d3bb1442784d3e4833b5efb84c3f3c221 not found: ID does not exist"
Jul 01 21:46:52.262485 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.262468    2573 scope.go:117] "RemoveContainer" containerID="12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c"
Jul 01 21:46:52.262674 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:46:52.262659    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c\": container with ID starting with 12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c not found: ID does not exist" containerID="12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c"
Jul 01 21:46:52.262716 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.262677    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c"} err="failed to get container status \"12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c\": rpc error: code = NotFound desc = could not find container \"12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c\": container with ID starting with 12f3573ccf814d6fac2d2697a2a8a9be5f69b4ab8415aef02d2ba9ab917b899c not found: ID does not exist"
Jul 01 21:46:52.293627 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.293605    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z24b5\" (UniqueName: \"kubernetes.io/projected/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kube-api-access-z24b5\") pod \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") "
Jul 01 21:46:52.293717 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.293650    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tmp-dir\") pod \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") "
Jul 01 21:46:52.293717 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.293668    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kserve-provision-location\") pod \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") "
Jul 01 21:46:52.293717 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.293690    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-home\") pod \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") "
Jul 01 21:46:52.293906 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.293718    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-dshm\") pod \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") "
Jul 01 21:46:52.293906 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.293738    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-model-cache\") pod \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") "
Jul 01 21:46:52.293906 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.293755    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tls-certs\") pod \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\" (UID: \"3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2\") "
Jul 01 21:46:52.294083 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.294052    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" (UID: "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:52.294385 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.294352    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-home" (OuterVolumeSpecName: "home") pod "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" (UID: "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:52.296630 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.296599    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-dshm" (OuterVolumeSpecName: "dshm") pod "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" (UID: "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:52.296730 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.296696    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kube-api-access-z24b5" (OuterVolumeSpecName: "kube-api-access-z24b5") pod "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" (UID: "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2"). InnerVolumeSpecName "kube-api-access-z24b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:46:52.296730 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.296714    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" (UID: "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:46:52.312795 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.312772    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" (UID: "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:52.360471 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.360443    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" (UID: "3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:46:52.394935 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.394914    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24b5\" (UniqueName: \"kubernetes.io/projected/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kube-api-access-z24b5\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:52.395012 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.394934    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:52.395012 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.394947    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:52.395012 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.394954    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:52.395012 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.394964    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:52.395012 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.394972    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:52.395012 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.394980    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:46:52.580730 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.580700    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 01 21:46:52.592724 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:52.592700    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 01 21:46:53.593894 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:53.593841    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" path="/var/lib/kubelet/pods/3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2/volumes"
Jul 01 21:46:57.271193 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:57.271163    2573 generic.go:356] "Generic (PLEG): container finished" podID="a994a194-21f7-4baf-a341-bf881923c36e" containerID="855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88" exitCode=0
Jul 01 21:46:57.271517 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:57.271233    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" event={"ID":"a994a194-21f7-4baf-a341-bf881923c36e","Type":"ContainerDied","Data":"855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88"}
Jul 01 21:46:58.278212 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:58.278177    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" event={"ID":"a994a194-21f7-4baf-a341-bf881923c36e","Type":"ContainerStarted","Data":"50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b"}
Jul 01 21:46:58.303146 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:58.303091    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" podStartSLOduration=2.290749541 podStartE2EDuration="14.303071746s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.012322205s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:46:58.300132643 +0000 UTC m=+3651.211467270" watchObservedRunningTime="2026-07-01 21:46:58.303071746 +0000 UTC m=+3651.214406373"
Jul 01 21:46:58.975489 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:46:58.975462    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:47:04.974951 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:04.974910    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:47:04.976634 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:04.976601    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 01 21:47:14.975249 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:14.975204    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 01 21:47:19.536798 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.536767    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"]
Jul 01 21:47:19.537178 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.537114    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" containerName="storage-initializer"
Jul 01 21:47:19.537178 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.537126    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" containerName="storage-initializer"
Jul 01 21:47:19.537178 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.537139    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" containerName="main"
Jul 01 21:47:19.537178 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.537144    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" containerName="main"
Jul 01 21:47:19.537314 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.537206    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3101b4e4-2fd9-4ce0-a351-1deca1cd1bc2" containerName="main"
Jul 01 21:47:19.540327 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.540309    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.544341 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.544322    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 01 21:47:19.544456 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.544378    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-z2g6t\""
Jul 01 21:47:19.550181 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.550160    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"]
Jul 01 21:47:19.619249 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.619226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br5w4\" (UniqueName: \"kubernetes.io/projected/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-kube-api-access-br5w4\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.619387 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.619262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.619387 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.619372    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.720663 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.720631    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.720814 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.720676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br5w4\" (UniqueName: \"kubernetes.io/projected/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-kube-api-access-br5w4\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.720814 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.720710    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.721042 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.721020    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.723131 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.723108    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.729370 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.729339    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br5w4\" (UniqueName: \"kubernetes.io/projected/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-kube-api-access-br5w4\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.850926 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.850898    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:19.986744 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:19.986704    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"]
Jul 01 21:47:19.989315 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:47:19.989288    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ab6db8_cdaf_441b_8ea4_8d3fa15c5941.slice/crio-61d7efc0e0a633869130c78fffd4d46b41f66f9d32716516f514dd7a03ea68c2 WatchSource:0}: Error finding container 61d7efc0e0a633869130c78fffd4d46b41f66f9d32716516f514dd7a03ea68c2: Status 404 returned error can't find the container with id 61d7efc0e0a633869130c78fffd4d46b41f66f9d32716516f514dd7a03ea68c2
Jul 01 21:47:20.376405 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:20.376373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs" event={"ID":"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941","Type":"ContainerStarted","Data":"435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b"}
Jul 01 21:47:20.376592 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:20.376425    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs" event={"ID":"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941","Type":"ContainerStarted","Data":"61d7efc0e0a633869130c78fffd4d46b41f66f9d32716516f514dd7a03ea68c2"}
Jul 01 21:47:20.376592 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:20.376460    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:20.406005 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:20.405958    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs" podStartSLOduration=1.405945505 podStartE2EDuration="1.405945505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:47:20.403549599 +0000 UTC m=+3673.314884228" watchObservedRunningTime="2026-07-01 21:47:20.405945505 +0000 UTC m=+3673.317280130"
Jul 01 21:47:24.975707 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:24.975621    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 01 21:47:34.975127 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:34.975042    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 01 21:47:44.975757 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:44.975713    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 01 21:47:51.382781 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:51.382750    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:52.483581 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.483546    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"]
Jul 01 21:47:52.493037 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:47:52.493006    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 21:47:52.493037 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:47:52.493037 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:47:52.493037 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b" cmd=["/bin/sleep","15"]
Jul 01 21:47:52.493362 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.493050    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs" message="" err=<
Jul 01 21:47:52.493362 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:47:52.493362 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:47:52.493362 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 21:47:52.493362 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:47:52.493063    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 21:47:52.493362 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:47:52.493362 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:47:52.493362 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs" podUID="04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" containerName="main" containerID="cri-o://435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b"
Jul 01 21:47:52.493362 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.493097    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs" podUID="04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" containerName="main" containerID="cri-o://435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b" gracePeriod=60
Jul 01 21:47:52.733391 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.733359    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:52.791540 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.791471    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tokenizer-uds\") pod \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") "
Jul 01 21:47:52.791540 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.791514    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tls-certs\") pod \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") "
Jul 01 21:47:52.791698 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.791586    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br5w4\" (UniqueName: \"kubernetes.io/projected/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-kube-api-access-br5w4\") pod \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\" (UID: \"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941\") "
Jul 01 21:47:52.791760 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.791740    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" (UID: "04ab6db8-cdaf-441b-8ea4-8d3fa15c5941"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:47:52.791809 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.791799    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:47:52.793654 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.793619    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" (UID: "04ab6db8-cdaf-441b-8ea4-8d3fa15c5941"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:47:52.793754 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.793724    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-kube-api-access-br5w4" (OuterVolumeSpecName: "kube-api-access-br5w4") pod "04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" (UID: "04ab6db8-cdaf-441b-8ea4-8d3fa15c5941"). InnerVolumeSpecName "kube-api-access-br5w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:47:52.892288 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.892257    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:47:52.892288 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:52.892283    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br5w4\" (UniqueName: \"kubernetes.io/projected/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941-kube-api-access-br5w4\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:47:53.534098 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.534061    2573 generic.go:356] "Generic (PLEG): container finished" podID="04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" containerID="435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b" exitCode=0
Jul 01 21:47:53.534565 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.534112    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs" event={"ID":"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941","Type":"ContainerDied","Data":"435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b"}
Jul 01 21:47:53.534565 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.534159    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs" event={"ID":"04ab6db8-cdaf-441b-8ea4-8d3fa15c5941","Type":"ContainerDied","Data":"61d7efc0e0a633869130c78fffd4d46b41f66f9d32716516f514dd7a03ea68c2"}
Jul 01 21:47:53.534565 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.534187    2573 scope.go:117] "RemoveContainer" containerID="435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b"
Jul 01 21:47:53.534565 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.534114    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"
Jul 01 21:47:53.542719 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.542701    2573 scope.go:117] "RemoveContainer" containerID="435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b"
Jul 01 21:47:53.543024 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:47:53.543004    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b\": container with ID starting with 435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b not found: ID does not exist" containerID="435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b"
Jul 01 21:47:53.543099 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.543032    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b"} err="failed to get container status \"435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b\": rpc error: code = NotFound desc = could not find container \"435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b\": container with ID starting with 435484bfc115cb1721280550d88a59f93fc60bdad6b28cc5c608c6fd0eafee2b not found: ID does not exist"
Jul 01 21:47:53.571438 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.571410    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"]
Jul 01 21:47:53.573087 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.573066    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs"]
Jul 01 21:47:53.591324 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:53.591294    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" path="/var/lib/kubelet/pods/04ab6db8-cdaf-441b-8ea4-8d3fa15c5941/volumes"
Jul 01 21:47:54.975215 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:47:54.975176    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 01 21:48:04.975880 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:04.975824    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 01 21:48:14.985660 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:14.985630    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:48:14.993289 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:14.993262    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:48:28.714535 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.714501    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"]
Jul 01 21:48:28.714940 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.714846    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" containerName="main"
Jul 01 21:48:28.714940 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.714873    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" containerName="main"
Jul 01 21:48:28.715025 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.714946    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ab6db8-cdaf-441b-8ea4-8d3fa15c5941" containerName="main"
Jul 01 21:48:28.716589 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.716573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:28.720622 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.720602    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 01 21:48:28.720745 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.720606    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-wr4v2\""
Jul 01 21:48:28.728480 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.728453    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"]
Jul 01 21:48:28.876549 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.876518    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:28.876696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.876626    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q66hd\" (UniqueName: \"kubernetes.io/projected/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-kube-api-access-q66hd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:28.876696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.876661    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:28.977151 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.977078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:28.977329 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.977171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q66hd\" (UniqueName: \"kubernetes.io/projected/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-kube-api-access-q66hd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:28.977329 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.977196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:28.977454 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.977443    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:28.979843 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.979817    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:28.986284 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:28.986262    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q66hd\" (UniqueName: \"kubernetes.io/projected/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-kube-api-access-q66hd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:29.026037 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:29.026012    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:29.163626 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:29.163603    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"]
Jul 01 21:48:29.166033 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:48:29.166007    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0d7dfa_90d5_4a53_a08d_d9dc2d7b1d37.slice/crio-98cdc3cd831ae3843a56ae66d8362724b1cb16e091282bb5d83e780a0aa198b8 WatchSource:0}: Error finding container 98cdc3cd831ae3843a56ae66d8362724b1cb16e091282bb5d83e780a0aa198b8: Status 404 returned error can't find the container with id 98cdc3cd831ae3843a56ae66d8362724b1cb16e091282bb5d83e780a0aa198b8
Jul 01 21:48:29.690030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:29.689991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z" event={"ID":"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37","Type":"ContainerStarted","Data":"1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea"}
Jul 01 21:48:29.690219 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:29.690043    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z" event={"ID":"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37","Type":"ContainerStarted","Data":"98cdc3cd831ae3843a56ae66d8362724b1cb16e091282bb5d83e780a0aa198b8"}
Jul 01 21:48:29.690219 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:29.690063    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:48:29.731284 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:48:29.731237    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z" podStartSLOduration=1.731222 podStartE2EDuration="1.731222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:48:29.73046284 +0000 UTC m=+3742.641797469" watchObservedRunningTime="2026-07-01 21:48:29.731222 +0000 UTC m=+3742.642556626"
Jul 01 21:49:00.696768 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:49:00.696675    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:51:07.901697 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:51:07.901601    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:51:07.905509 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:51:07.905417    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:51:07.908613 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:51:07.908595    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:51:07.912518 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:51:07.912496    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:54:39.305045 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:39.305014    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"]
Jul 01 21:54:54.328912 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:54.328834    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main" containerID="cri-o://50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b" gracePeriod=45
Jul 01 21:54:54.897246 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:54.897221    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:54:55.063041 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.062953    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-home\") pod \"a994a194-21f7-4baf-a341-bf881923c36e\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") "
Jul 01 21:54:55.063041 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.062990    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8274n\" (UniqueName: \"kubernetes.io/projected/a994a194-21f7-4baf-a341-bf881923c36e-kube-api-access-8274n\") pod \"a994a194-21f7-4baf-a341-bf881923c36e\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") "
Jul 01 21:54:55.063041 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.063009    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-kserve-provision-location\") pod \"a994a194-21f7-4baf-a341-bf881923c36e\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") "
Jul 01 21:54:55.063041 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.063032    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-model-cache\") pod \"a994a194-21f7-4baf-a341-bf881923c36e\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") "
Jul 01 21:54:55.063397 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.063058    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-dshm\") pod \"a994a194-21f7-4baf-a341-bf881923c36e\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") "
Jul 01 21:54:55.063397 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.063147    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a994a194-21f7-4baf-a341-bf881923c36e-tls-certs\") pod \"a994a194-21f7-4baf-a341-bf881923c36e\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") "
Jul 01 21:54:55.063397 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.063185    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-tmp-dir\") pod \"a994a194-21f7-4baf-a341-bf881923c36e\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") "
Jul 01 21:54:55.063397 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.063242    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a994a194-21f7-4baf-a341-bf881923c36e-cabundle-cert\") pod \"a994a194-21f7-4baf-a341-bf881923c36e\" (UID: \"a994a194-21f7-4baf-a341-bf881923c36e\") "
Jul 01 21:54:55.063619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.063428    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a994a194-21f7-4baf-a341-bf881923c36e" (UID: "a994a194-21f7-4baf-a341-bf881923c36e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:54:55.063619 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.063537    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-home" (OuterVolumeSpecName: "home") pod "a994a194-21f7-4baf-a341-bf881923c36e" (UID: "a994a194-21f7-4baf-a341-bf881923c36e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:54:55.063878 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.063766    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a994a194-21f7-4baf-a341-bf881923c36e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a994a194-21f7-4baf-a341-bf881923c36e" (UID: "a994a194-21f7-4baf-a341-bf881923c36e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:54:55.065467 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.065436    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a994a194-21f7-4baf-a341-bf881923c36e-kube-api-access-8274n" (OuterVolumeSpecName: "kube-api-access-8274n") pod "a994a194-21f7-4baf-a341-bf881923c36e" (UID: "a994a194-21f7-4baf-a341-bf881923c36e"). InnerVolumeSpecName "kube-api-access-8274n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:54:55.065579 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.065517    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a994a194-21f7-4baf-a341-bf881923c36e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a994a194-21f7-4baf-a341-bf881923c36e" (UID: "a994a194-21f7-4baf-a341-bf881923c36e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:54:55.065708 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.065685    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-dshm" (OuterVolumeSpecName: "dshm") pod "a994a194-21f7-4baf-a341-bf881923c36e" (UID: "a994a194-21f7-4baf-a341-bf881923c36e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:54:55.078536 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.078512    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a994a194-21f7-4baf-a341-bf881923c36e" (UID: "a994a194-21f7-4baf-a341-bf881923c36e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:54:55.134994 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.134963    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a994a194-21f7-4baf-a341-bf881923c36e" (UID: "a994a194-21f7-4baf-a341-bf881923c36e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:54:55.163848 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.163824    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a994a194-21f7-4baf-a341-bf881923c36e-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:54:55.163848 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.163844    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:54:55.164020 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.163873    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a994a194-21f7-4baf-a341-bf881923c36e-cabundle-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:54:55.164020 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.163883    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:54:55.164020 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.163891    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8274n\" (UniqueName: \"kubernetes.io/projected/a994a194-21f7-4baf-a341-bf881923c36e-kube-api-access-8274n\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:54:55.164020 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.163900    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:54:55.164020 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.163909    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:54:55.164020 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.163918    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a994a194-21f7-4baf-a341-bf881923c36e-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:54:55.388168 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.388139    2573 generic.go:356] "Generic (PLEG): container finished" podID="a994a194-21f7-4baf-a341-bf881923c36e" containerID="50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b" exitCode=0
Jul 01 21:54:55.388532 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.388195    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"
Jul 01 21:54:55.388532 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.388217    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" event={"ID":"a994a194-21f7-4baf-a341-bf881923c36e","Type":"ContainerDied","Data":"50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b"}
Jul 01 21:54:55.388532 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.388273    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2" event={"ID":"a994a194-21f7-4baf-a341-bf881923c36e","Type":"ContainerDied","Data":"a2aa0d035c59f1d5e9729124cea9e69f04df142983decf4bf3ea1f59daeea611"}
Jul 01 21:54:55.388532 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.388295    2573 scope.go:117] "RemoveContainer" containerID="50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b"
Jul 01 21:54:55.397525 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.397505    2573 scope.go:117] "RemoveContainer" containerID="855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88"
Jul 01 21:54:55.405149 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.405132    2573 scope.go:117] "RemoveContainer" containerID="50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b"
Jul 01 21:54:55.405363 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:54:55.405347    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b\": container with ID starting with 50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b not found: ID does not exist" containerID="50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b"
Jul 01 21:54:55.405417 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.405372    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b"} err="failed to get container status \"50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b\": rpc error: code = NotFound desc = could not find container \"50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b\": container with ID starting with 50f2fd3a8cb482c58b38c245dd91db0718237e79021bb64021c90095fead217b not found: ID does not exist"
Jul 01 21:54:55.405417 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.405391    2573 scope.go:117] "RemoveContainer" containerID="855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88"
Jul 01 21:54:55.405630 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:54:55.405613    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88\": container with ID starting with 855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88 not found: ID does not exist" containerID="855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88"
Jul 01 21:54:55.405671 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.405634    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88"} err="failed to get container status \"855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88\": rpc error: code = NotFound desc = could not find container \"855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88\": container with ID starting with 855ab77bf9edf834ef0888d9d4f8ea85e9b0711ba28d16f90e0ce3b1c874ac88 not found: ID does not exist"
Jul 01 21:54:55.411270 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.411251    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"]
Jul 01 21:54:55.414480 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.414461    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2"]
Jul 01 21:54:55.590720 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:54:55.590691    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a994a194-21f7-4baf-a341-bf881923c36e" path="/var/lib/kubelet/pods/a994a194-21f7-4baf-a341-bf881923c36e/volumes"
Jul 01 21:55:15.198054 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.198019    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"]
Jul 01 21:55:15.198460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.198333    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main"
Jul 01 21:55:15.198460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.198344    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main"
Jul 01 21:55:15.198460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.198354    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="storage-initializer"
Jul 01 21:55:15.198460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.198359    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="storage-initializer"
Jul 01 21:55:15.198460 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.198409    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a994a194-21f7-4baf-a341-bf881923c36e" containerName="main"
Jul 01 21:55:15.201371 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.201353    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.204359 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.204331    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:55:15.204359 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.204339    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 01 21:55:15.213037 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.213015    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"]
Jul 01 21:55:15.312055 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.312028    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kserve-provision-location\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.312195 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.312059    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-dshm\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.312195 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.312093    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-home\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.312195 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.312159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-model-cache\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.312348 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.312217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tls-certs\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.312348 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.312237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tmp-dir\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.312348 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.312252    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctf8z\" (UniqueName: \"kubernetes.io/projected/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kube-api-access-ctf8z\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.412817 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.412785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-model-cache\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.412973 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.412826    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tls-certs\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.412973 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.412846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tmp-dir\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.413088 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.412983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctf8z\" (UniqueName: \"kubernetes.io/projected/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kube-api-access-ctf8z\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.413088 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.413075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kserve-provision-location\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.413188 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.413117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-dshm\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.413188 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.413183    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-home\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.413287 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.413215    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-model-cache\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.413287 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.413185    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tmp-dir\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.413416 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.413393    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kserve-provision-location\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.413467 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.413405    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-home\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.415147 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.415124    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-dshm\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.415422 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.415406    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tls-certs\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.422296 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.422271    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctf8z\" (UniqueName: \"kubernetes.io/projected/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kube-api-access-ctf8z\") pod \"tls-verification-test-kserve-654fdf69fb-mc965\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.511911 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.511833    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:15.646254 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:15.646230    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"]
Jul 01 21:55:15.648580 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:55:15.648551    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b00e2d_1311_41ba_ad1b_d3086ff9c0b8.slice/crio-3b39d0eee42b7731ea322a601b8dc0b44701164b2f8f5838f9391f831c8a1148 WatchSource:0}: Error finding container 3b39d0eee42b7731ea322a601b8dc0b44701164b2f8f5838f9391f831c8a1148: Status 404 returned error can't find the container with id 3b39d0eee42b7731ea322a601b8dc0b44701164b2f8f5838f9391f831c8a1148
Jul 01 21:55:16.484369 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:16.484329    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" event={"ID":"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8","Type":"ContainerStarted","Data":"89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6"}
Jul 01 21:55:16.484802 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:16.484391    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" event={"ID":"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8","Type":"ContainerStarted","Data":"3b39d0eee42b7731ea322a601b8dc0b44701164b2f8f5838f9391f831c8a1148"}
Jul 01 21:55:20.509074 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:20.509038    2573 generic.go:356] "Generic (PLEG): container finished" podID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerID="89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6" exitCode=0
Jul 01 21:55:20.509476 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:20.509113    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" event={"ID":"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8","Type":"ContainerDied","Data":"89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6"}
Jul 01 21:55:21.516154 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:21.516115    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" event={"ID":"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8","Type":"ContainerStarted","Data":"e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9"}
Jul 01 21:55:21.543454 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:21.543401    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podStartSLOduration=2.407036267 podStartE2EDuration="6.543385354s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.136349087s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:55:21.543316622 +0000 UTC m=+4154.454651250" watchObservedRunningTime="2026-07-01 21:55:21.543385354 +0000 UTC m=+4154.454719979"
Jul 01 21:55:22.512932 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:22.512898    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:25.512696 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:25.512657    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:55:25.514294 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:25.514267    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:55:35.512812 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:35.512758    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:55:41.568950 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.568917    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"]
Jul 01 21:55:41.577920 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:55:41.577883    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 21:55:41.577920 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:55:41.577920 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:55:41.577920 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea" cmd=["/bin/sleep","15"]
Jul 01 21:55:41.578213 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.577939    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z" message="" err=<
Jul 01 21:55:41.578213 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:55:41.578213 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:55:41.578213 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 21:55:41.578213 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:55:41.577955    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 21:55:41.578213 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 21:55:41.578213 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 21:55:41.578213 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z" podUID="ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" containerName="main" containerID="cri-o://1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea"
Jul 01 21:55:41.578213 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.578013    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z" podUID="ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" containerName="main" containerID="cri-o://1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea" gracePeriod=60
Jul 01 21:55:41.825407 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.825343    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:55:41.936177 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.936135    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tokenizer-uds\") pod \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") "
Jul 01 21:55:41.936338 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.936198    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tls-certs\") pod \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") "
Jul 01 21:55:41.936338 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.936248    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q66hd\" (UniqueName: \"kubernetes.io/projected/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-kube-api-access-q66hd\") pod \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\" (UID: \"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37\") "
Jul 01 21:55:41.936476 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.936447    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" (UID: "ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:55:41.938323 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.938289    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" (UID: "ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:55:41.938323 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:41.938318    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-kube-api-access-q66hd" (OuterVolumeSpecName: "kube-api-access-q66hd") pod "ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" (UID: "ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37"). InnerVolumeSpecName "kube-api-access-q66hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:55:42.037153 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.037119    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:55:42.037153 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.037147    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:55:42.037153 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.037156    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q66hd\" (UniqueName: \"kubernetes.io/projected/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37-kube-api-access-q66hd\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:55:42.613448 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.613415    2573 generic.go:356] "Generic (PLEG): container finished" podID="ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" containerID="1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea" exitCode=0
Jul 01 21:55:42.613898 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.613467    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"
Jul 01 21:55:42.613898 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.613470    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z" event={"ID":"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37","Type":"ContainerDied","Data":"1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea"}
Jul 01 21:55:42.613898 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.613610    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z" event={"ID":"ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37","Type":"ContainerDied","Data":"98cdc3cd831ae3843a56ae66d8362724b1cb16e091282bb5d83e780a0aa198b8"}
Jul 01 21:55:42.613898 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.613648    2573 scope.go:117] "RemoveContainer" containerID="1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea"
Jul 01 21:55:42.622476 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.622461    2573 scope.go:117] "RemoveContainer" containerID="1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea"
Jul 01 21:55:42.622730 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:55:42.622710    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea\": container with ID starting with 1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea not found: ID does not exist" containerID="1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea"
Jul 01 21:55:42.622804 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.622736    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea"} err="failed to get container status \"1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea\": rpc error: code = NotFound desc = could not find container \"1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea\": container with ID starting with 1fbd535e86ce9da211e2012b1c905c026ac125bc45f49f7d471a3d72886a3fea not found: ID does not exist"
Jul 01 21:55:42.637960 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.637929    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"]
Jul 01 21:55:42.641880 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:42.641840    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z"]
Jul 01 21:55:43.592730 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:43.592697    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" path="/var/lib/kubelet/pods/ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37/volumes"
Jul 01 21:55:45.512531 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:45.512493    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:55:55.513000 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:55:55.512952    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:56:05.102329 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.102293    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"]
Jul 01 21:56:05.102790 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.102771    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" containerName="main"
Jul 01 21:56:05.102844 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.102797    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" containerName="main"
Jul 01 21:56:05.102904 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.102897    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef0d7dfa-90d5-4a53-a08d-d9dc2d7b1d37" containerName="main"
Jul 01 21:56:05.105106 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.105088    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.109292 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.109269    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 01 21:56:05.109427 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.109307    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-sxbh6\""
Jul 01 21:56:05.115467 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.115443    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"]
Jul 01 21:56:05.122073 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.122051    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6qgq\" (UniqueName: \"kubernetes.io/projected/5dedf8a4-8117-469a-bd30-34c8a2173219-kube-api-access-b6qgq\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.122173 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.122091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dedf8a4-8117-469a-bd30-34c8a2173219-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.122173 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.122131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5dedf8a4-8117-469a-bd30-34c8a2173219-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.223442 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.223414    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6qgq\" (UniqueName: \"kubernetes.io/projected/5dedf8a4-8117-469a-bd30-34c8a2173219-kube-api-access-b6qgq\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.223561 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.223458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dedf8a4-8117-469a-bd30-34c8a2173219-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.223561 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.223519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5dedf8a4-8117-469a-bd30-34c8a2173219-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.223930 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.223913    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5dedf8a4-8117-469a-bd30-34c8a2173219-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.226004 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.225978    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dedf8a4-8117-469a-bd30-34c8a2173219-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.233639 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.233612    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6qgq\" (UniqueName: \"kubernetes.io/projected/5dedf8a4-8117-469a-bd30-34c8a2173219-kube-api-access-b6qgq\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.416877 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.416775    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.512831 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.512766    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:56:05.551588 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.551565    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"]
Jul 01 21:56:05.554077 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:56:05.554052    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dedf8a4_8117_469a_bd30_34c8a2173219.slice/crio-835caef2341896505f8602651c066e1f2e0372215031d5bab12977cd111fd2d1 WatchSource:0}: Error finding container 835caef2341896505f8602651c066e1f2e0372215031d5bab12977cd111fd2d1: Status 404 returned error can't find the container with id 835caef2341896505f8602651c066e1f2e0372215031d5bab12977cd111fd2d1
Jul 01 21:56:05.722383 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.722301    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6" event={"ID":"5dedf8a4-8117-469a-bd30-34c8a2173219","Type":"ContainerStarted","Data":"5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b"}
Jul 01 21:56:05.722383 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.722361    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6" event={"ID":"5dedf8a4-8117-469a-bd30-34c8a2173219","Type":"ContainerStarted","Data":"835caef2341896505f8602651c066e1f2e0372215031d5bab12977cd111fd2d1"}
Jul 01 21:56:05.722566 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.722427    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:05.743563 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:05.743512    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6" podStartSLOduration=0.743498889 podStartE2EDuration="743.498889ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:56:05.742845546 +0000 UTC m=+4198.654180172" watchObservedRunningTime="2026-07-01 21:56:05.743498889 +0000 UTC m=+4198.654833514"
Jul 01 21:56:07.929165 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:07.929047    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:56:07.938343 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:07.935183    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 21:56:07.938343 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:07.936568    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:56:07.942530 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:07.942507    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 21:56:15.513132 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:15.513085    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:56:25.512346 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:25.512304    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:56:35.512432 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:35.512349    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:56:36.727664 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:36.727629    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 21:56:45.513066 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:45.513024    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:56:55.513030 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:56:55.512985    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:57:05.513171 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:05.513130    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:57:15.522560 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:15.522529    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:57:15.530286 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:15.530258    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:57:19.875578 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:19.875539    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"]
Jul 01 21:57:34.899681 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:34.899640    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" containerID="cri-o://e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9" gracePeriod=45
Jul 01 21:57:35.524045 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.524008    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 01 21:57:35.647230 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.647205    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:57:35.784963 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.784886    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tmp-dir\") pod \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") "
Jul 01 21:57:35.784963 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.784927    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tls-certs\") pod \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") "
Jul 01 21:57:35.784963 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.784956    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kserve-provision-location\") pod \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") "
Jul 01 21:57:35.785242 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.784995    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-home\") pod \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") "
Jul 01 21:57:35.785242 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.785014    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-dshm\") pod \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") "
Jul 01 21:57:35.785242 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.785029    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-model-cache\") pod \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") "
Jul 01 21:57:35.785242 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.785053    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctf8z\" (UniqueName: \"kubernetes.io/projected/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kube-api-access-ctf8z\") pod \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\" (UID: \"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8\") "
Jul 01 21:57:35.785456 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.785405    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" (UID: "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:57:35.785679 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.785654    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-home" (OuterVolumeSpecName: "home") pod "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" (UID: "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:57:35.787330 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.787306    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" (UID: "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:57:35.787433 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.787324    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kube-api-access-ctf8z" (OuterVolumeSpecName: "kube-api-access-ctf8z") pod "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" (UID: "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8"). InnerVolumeSpecName "kube-api-access-ctf8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:57:35.787433 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.787407    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-dshm" (OuterVolumeSpecName: "dshm") pod "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" (UID: "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:57:35.804425 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.804393    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" (UID: "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:57:35.861374 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.861343    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" (UID: "87b00e2d-1311-41ba-ad1b-d3086ff9c0b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:57:35.885848 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.885823    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:57:35.885848 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.885845    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:57:35.885848 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.885867    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:57:35.886058 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.885876    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:57:35.886058 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.885886    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:57:35.886058 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.885893    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:57:35.886058 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:35.885901    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctf8z\" (UniqueName: \"kubernetes.io/projected/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8-kube-api-access-ctf8z\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 21:57:36.138255 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.138230    2573 generic.go:356] "Generic (PLEG): container finished" podID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerID="e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9" exitCode=0
Jul 01 21:57:36.138609 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.138279    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"
Jul 01 21:57:36.138609 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.138319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" event={"ID":"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8","Type":"ContainerDied","Data":"e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9"}
Jul 01 21:57:36.138609 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.138376    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965" event={"ID":"87b00e2d-1311-41ba-ad1b-d3086ff9c0b8","Type":"ContainerDied","Data":"3b39d0eee42b7731ea322a601b8dc0b44701164b2f8f5838f9391f831c8a1148"}
Jul 01 21:57:36.138609 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.138406    2573 scope.go:117] "RemoveContainer" containerID="e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9"
Jul 01 21:57:36.147645 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.147627    2573 scope.go:117] "RemoveContainer" containerID="89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6"
Jul 01 21:57:36.155211 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.155196    2573 scope.go:117] "RemoveContainer" containerID="e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9"
Jul 01 21:57:36.155432 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:57:36.155412    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9\": container with ID starting with e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9 not found: ID does not exist" containerID="e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9"
Jul 01 21:57:36.155488 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.155440    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9"} err="failed to get container status \"e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9\": rpc error: code = NotFound desc = could not find container \"e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9\": container with ID starting with e228ccff4e70aa724e755cc7974bc9541e8b332d00b689c42f09cc3969c52da9 not found: ID does not exist"
Jul 01 21:57:36.155488 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.155456    2573 scope.go:117] "RemoveContainer" containerID="89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6"
Jul 01 21:57:36.155686 ip-10-0-138-219 kubenswrapper[2573]: E0701 21:57:36.155670    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6\": container with ID starting with 89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6 not found: ID does not exist" containerID="89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6"
Jul 01 21:57:36.155733 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.155689    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6"} err="failed to get container status \"89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6\": rpc error: code = NotFound desc = could not find container \"89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6\": container with ID starting with 89c663b57773c1b902ae7a1302fcd89ef6338d66f364cf144e65500e418a38c6 not found: ID does not exist"
Jul 01 21:57:36.161156 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.161135    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"]
Jul 01 21:57:36.164920 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:36.164901    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965"]
Jul 01 21:57:37.591207 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:37.591174    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" path="/var/lib/kubelet/pods/87b00e2d-1311-41ba-ad1b-d3086ff9c0b8/volumes"
Jul 01 21:57:45.702902 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.702844    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"]
Jul 01 21:57:45.703391 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.703194    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="storage-initializer"
Jul 01 21:57:45.703391 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.703206    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="storage-initializer"
Jul 01 21:57:45.703391 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.703216    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main"
Jul 01 21:57:45.703391 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.703222    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main"
Jul 01 21:57:45.703391 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.703281    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b00e2d-1311-41ba-ad1b-d3086ff9c0b8" containerName="main"
Jul 01 21:57:45.707994 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.707978    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.710766 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.710746    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 21:57:45.710925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.710787    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 01 21:57:45.717496 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.717468    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"]
Jul 01 21:57:45.859446 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.859413    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhm2w\" (UniqueName: \"kubernetes.io/projected/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kube-api-access-rhm2w\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.859622 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.859454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tmp-dir\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.859622 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.859507    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tls-certs\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.859622 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.859554    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kserve-provision-location\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.859622 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.859601    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-dshm\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.859818 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.859627    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-model-cache\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.859818 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.859702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-home\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961046 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.960964    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-home\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961046 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961020    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhm2w\" (UniqueName: \"kubernetes.io/projected/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kube-api-access-rhm2w\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961219 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961124    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tmp-dir\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961219 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tls-certs\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961324 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kserve-provision-location\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961324 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-dshm\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961324 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-model-cache\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961461 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961368    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-home\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961461 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961413    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tmp-dir\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961626 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961604    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-model-cache\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.961693 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.961630    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kserve-provision-location\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.963355 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.963336    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-dshm\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.963748 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.963730    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tls-certs\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:45.970922 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:45.970903    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhm2w\" (UniqueName: \"kubernetes.io/projected/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kube-api-access-rhm2w\") pod \"prestop-hook-test-kserve-6769cdd744-b8f4z\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:46.018636 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:46.018618    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:46.148057 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:46.148033    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"]
Jul 01 21:57:46.150075 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:57:46.150047    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfe85a7_7665_4391_adf3_ebd19a2a0c48.slice/crio-5cdd61f6452a129878fea7b87a756463922f2f0c377b4d7c5ecbe4e3f13aa776 WatchSource:0}: Error finding container 5cdd61f6452a129878fea7b87a756463922f2f0c377b4d7c5ecbe4e3f13aa776: Status 404 returned error can't find the container with id 5cdd61f6452a129878fea7b87a756463922f2f0c377b4d7c5ecbe4e3f13aa776
Jul 01 21:57:46.186334 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:46.186310    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" event={"ID":"8dfe85a7-7665-4391-adf3-ebd19a2a0c48","Type":"ContainerStarted","Data":"5cdd61f6452a129878fea7b87a756463922f2f0c377b4d7c5ecbe4e3f13aa776"}
Jul 01 21:57:47.191912 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:47.191848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" event={"ID":"8dfe85a7-7665-4391-adf3-ebd19a2a0c48","Type":"ContainerStarted","Data":"43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389"}
Jul 01 21:57:51.210552 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:51.210517    2573 generic.go:356] "Generic (PLEG): container finished" podID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerID="43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389" exitCode=0
Jul 01 21:57:51.211043 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:51.210582    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" event={"ID":"8dfe85a7-7665-4391-adf3-ebd19a2a0c48","Type":"ContainerDied","Data":"43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389"}
Jul 01 21:57:52.216826 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:52.216796    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" event={"ID":"8dfe85a7-7665-4391-adf3-ebd19a2a0c48","Type":"ContainerStarted","Data":"260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818"}
Jul 01 21:57:52.243005 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:52.242956    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podStartSLOduration=3.257702151 podStartE2EDuration="7.242937291s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.98523514s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:57:52.238471641 +0000 UTC m=+4305.149806267" watchObservedRunningTime="2026-07-01 21:57:52.242937291 +0000 UTC m=+4305.154271917"
Jul 01 21:57:53.018932 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:53.018902    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:56.018798 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:56.018757    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:57:56.020247 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:57:56.020218    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:58:06.019749 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:58:06.019665    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:58:16.019980 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:58:16.019933    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:58:26.019982 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:58:26.019944    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:58:36.019237 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:58:36.019194    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:58:46.019477 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:58:46.019424    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:58:56.019177 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:58:56.019125    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:59:06.019784 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:06.019743    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:59:16.019720 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:16.019673    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:59:26.019670 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:26.019632    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:59:36.019088 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:36.019007    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 21:59:46.029089 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:46.029054    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:59:46.036816 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:46.036789    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 21:59:50.818623 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.818589    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"]
Jul 01 21:59:50.846398 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.846368    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"]
Jul 01 21:59:50.849386 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.849365    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.860764 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.860744    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"]
Jul 01 21:59:50.882527 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.882501    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-tmp-dir\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.882618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.882530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-dshm\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.882618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.882550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-home\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.882618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.882579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-kserve-provision-location\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.882618 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.882609    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n862\" (UniqueName: \"kubernetes.io/projected/12986664-524e-49c5-af92-2d10265930ea-kube-api-access-7n862\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.882777 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.882657    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-model-cache\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.882777 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.882712    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12986664-524e-49c5-af92-2d10265930ea-tls-certs\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.983765 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.983738    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-model-cache\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.983925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.983801    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12986664-524e-49c5-af92-2d10265930ea-tls-certs\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.983925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.983830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-tmp-dir\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.983925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.983846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-dshm\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.983925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.983880    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-home\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.983925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.983897    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-kserve-provision-location\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.983925 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.983918    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n862\" (UniqueName: \"kubernetes.io/projected/12986664-524e-49c5-af92-2d10265930ea-kube-api-access-7n862\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.984256 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.984157    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-model-cache\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.984313 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.984256    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-tmp-dir\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.984313 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.984297    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-home\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.984388 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.984371    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-kserve-provision-location\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.986207 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.986189    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-dshm\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.986386 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.986368    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12986664-524e-49c5-af92-2d10265930ea-tls-certs\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:50.991410 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:50.991389    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n862\" (UniqueName: \"kubernetes.io/projected/12986664-524e-49c5-af92-2d10265930ea-kube-api-access-7n862\") pod \"prestop-hook-test-kserve-6769cdd744-kpvrw\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:51.158820 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:51.158759    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 21:59:51.291050 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:51.291025    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"]
Jul 01 21:59:51.291661 ip-10-0-138-219 kubenswrapper[2573]: W0701 21:59:51.291633    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12986664_524e_49c5_af92_2d10265930ea.slice/crio-f8b537cb43adb22b8c4897611dff6f2af96a6955d1831c9a525b084659d665c4 WatchSource:0}: Error finding container f8b537cb43adb22b8c4897611dff6f2af96a6955d1831c9a525b084659d665c4: Status 404 returned error can't find the container with id f8b537cb43adb22b8c4897611dff6f2af96a6955d1831c9a525b084659d665c4
Jul 01 21:59:51.762646 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:51.762609    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" event={"ID":"12986664-524e-49c5-af92-2d10265930ea","Type":"ContainerStarted","Data":"b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c"}
Jul 01 21:59:51.762820 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:51.762665    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" event={"ID":"12986664-524e-49c5-af92-2d10265930ea","Type":"ContainerStarted","Data":"f8b537cb43adb22b8c4897611dff6f2af96a6955d1831c9a525b084659d665c4"}
Jul 01 21:59:55.782837 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:55.782805    2573 generic.go:356] "Generic (PLEG): container finished" podID="12986664-524e-49c5-af92-2d10265930ea" containerID="b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c" exitCode=0
Jul 01 21:59:55.783229 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:55.782882    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" event={"ID":"12986664-524e-49c5-af92-2d10265930ea","Type":"ContainerDied","Data":"b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c"}
Jul 01 21:59:56.789032 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:56.788995    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" event={"ID":"12986664-524e-49c5-af92-2d10265930ea","Type":"ContainerStarted","Data":"61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293"}
Jul 01 21:59:56.812056 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:56.812010    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" podStartSLOduration=2.992706692 podStartE2EDuration="6.811989995s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.819283303s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:59:56.809516766 +0000 UTC m=+4429.720851391" watchObservedRunningTime="2026-07-01 21:59:56.811989995 +0000 UTC m=+4429.723324624"
Jul 01 21:59:57.159250 ip-10-0-138-219 kubenswrapper[2573]: I0701 21:59:57.159213    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 22:00:01.159136 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:01.159098    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 22:00:01.160608 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:01.160577    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" podUID="12986664-524e-49c5-af92-2d10265930ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 01 22:00:05.841346 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:05.841298    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" containerID="cri-o://260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818" gracePeriod=45
Jul 01 22:00:06.030315 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.030268    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 01 22:00:06.625117 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.624718    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 22:00:06.724528 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.724502    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-model-cache\") pod \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") "
Jul 01 22:00:06.724705 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.724601    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-home\") pod \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") "
Jul 01 22:00:06.724705 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.724652    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhm2w\" (UniqueName: \"kubernetes.io/projected/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kube-api-access-rhm2w\") pod \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") "
Jul 01 22:00:06.724833 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.724752    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dfe85a7-7665-4391-adf3-ebd19a2a0c48" (UID: "8dfe85a7-7665-4391-adf3-ebd19a2a0c48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:06.724833 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.724684    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kserve-provision-location\") pod \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") "
Jul 01 22:00:06.724833 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.724810    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tls-certs\") pod \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") "
Jul 01 22:00:06.725035 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.724834    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tmp-dir\") pod \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") "
Jul 01 22:00:06.725035 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.724880    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-dshm\") pod \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\" (UID: \"8dfe85a7-7665-4391-adf3-ebd19a2a0c48\") "
Jul 01 22:00:06.725231 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.725212    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:06.725336 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.725311    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-home" (OuterVolumeSpecName: "home") pod "8dfe85a7-7665-4391-adf3-ebd19a2a0c48" (UID: "8dfe85a7-7665-4391-adf3-ebd19a2a0c48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:06.727258 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.727223    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kube-api-access-rhm2w" (OuterVolumeSpecName: "kube-api-access-rhm2w") pod "8dfe85a7-7665-4391-adf3-ebd19a2a0c48" (UID: "8dfe85a7-7665-4391-adf3-ebd19a2a0c48"). InnerVolumeSpecName "kube-api-access-rhm2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:00:06.728001 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.727973    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-dshm" (OuterVolumeSpecName: "dshm") pod "8dfe85a7-7665-4391-adf3-ebd19a2a0c48" (UID: "8dfe85a7-7665-4391-adf3-ebd19a2a0c48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:06.728119 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.728058    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dfe85a7-7665-4391-adf3-ebd19a2a0c48" (UID: "8dfe85a7-7665-4391-adf3-ebd19a2a0c48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 22:00:06.738685 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.738652    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dfe85a7-7665-4391-adf3-ebd19a2a0c48" (UID: "8dfe85a7-7665-4391-adf3-ebd19a2a0c48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:06.807900 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.807863    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8dfe85a7-7665-4391-adf3-ebd19a2a0c48" (UID: "8dfe85a7-7665-4391-adf3-ebd19a2a0c48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:06.826731 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.826700    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:06.826871 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.826736    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhm2w\" (UniqueName: \"kubernetes.io/projected/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kube-api-access-rhm2w\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:06.826871 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.826753    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:06.826871 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.826771    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:06.826871 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.826789    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:06.826871 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.826801    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dfe85a7-7665-4391-adf3-ebd19a2a0c48-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:06.836997 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.836963    2573 generic.go:356] "Generic (PLEG): container finished" podID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerID="260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818" exitCode=0
Jul 01 22:00:06.837097 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.837018    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"
Jul 01 22:00:06.837097 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.837042    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" event={"ID":"8dfe85a7-7665-4391-adf3-ebd19a2a0c48","Type":"ContainerDied","Data":"260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818"}
Jul 01 22:00:06.837172 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.837102    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z" event={"ID":"8dfe85a7-7665-4391-adf3-ebd19a2a0c48","Type":"ContainerDied","Data":"5cdd61f6452a129878fea7b87a756463922f2f0c377b4d7c5ecbe4e3f13aa776"}
Jul 01 22:00:06.837172 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.837125    2573 scope.go:117] "RemoveContainer" containerID="260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818"
Jul 01 22:00:06.846773 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.846527    2573 scope.go:117] "RemoveContainer" containerID="43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389"
Jul 01 22:00:06.855035 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.855017    2573 scope.go:117] "RemoveContainer" containerID="260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818"
Jul 01 22:00:06.855293 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:00:06.855268    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818\": container with ID starting with 260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818 not found: ID does not exist" containerID="260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818"
Jul 01 22:00:06.855378 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.855304    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818"} err="failed to get container status \"260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818\": rpc error: code = NotFound desc = could not find container \"260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818\": container with ID starting with 260cc268104f134c23b710cc364b377e08cb9291c92563594b9cfaca7b7a3818 not found: ID does not exist"
Jul 01 22:00:06.855378 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.855337    2573 scope.go:117] "RemoveContainer" containerID="43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389"
Jul 01 22:00:06.855590 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:00:06.855571    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389\": container with ID starting with 43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389 not found: ID does not exist" containerID="43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389"
Jul 01 22:00:06.855638 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.855596    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389"} err="failed to get container status \"43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389\": rpc error: code = NotFound desc = could not find container \"43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389\": container with ID starting with 43be39b50fc541712eb6815dd8c0a6b01b8ad1b0c04a2dbaf743856a5729c389 not found: ID does not exist"
Jul 01 22:00:06.859477 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.859457    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"]
Jul 01 22:00:06.865315 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:06.865271    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z"]
Jul 01 22:00:07.593168 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:07.593135    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" path="/var/lib/kubelet/pods/8dfe85a7-7665-4391-adf3-ebd19a2a0c48/volumes"
Jul 01 22:00:11.159618 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:11.159574    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" podUID="12986664-524e-49c5-af92-2d10265930ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 01 22:00:21.159674 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:21.159627    2573 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" podUID="12986664-524e-49c5-af92-2d10265930ea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 01 22:00:23.496477 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:23.496441    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"]
Jul 01 22:00:38.521307 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:38.521246    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" podUID="12986664-524e-49c5-af92-2d10265930ea" containerName="main" containerID="cri-o://61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293" gracePeriod=45
Jul 01 22:00:40.269061 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.269037    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-6769cdd744-kpvrw_12986664-524e-49c5-af92-2d10265930ea/main/0.log"
Jul 01 22:00:40.270570 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.270552    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 22:00:40.301887 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.301846    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-model-cache\") pod \"12986664-524e-49c5-af92-2d10265930ea\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") "
Jul 01 22:00:40.301998 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.301918    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-dshm\") pod \"12986664-524e-49c5-af92-2d10265930ea\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") "
Jul 01 22:00:40.301998 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.301939    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12986664-524e-49c5-af92-2d10265930ea-tls-certs\") pod \"12986664-524e-49c5-af92-2d10265930ea\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") "
Jul 01 22:00:40.301998 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.301963    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-tmp-dir\") pod \"12986664-524e-49c5-af92-2d10265930ea\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") "
Jul 01 22:00:40.302162 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.302001    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n862\" (UniqueName: \"kubernetes.io/projected/12986664-524e-49c5-af92-2d10265930ea-kube-api-access-7n862\") pod \"12986664-524e-49c5-af92-2d10265930ea\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") "
Jul 01 22:00:40.302162 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.302027    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-kserve-provision-location\") pod \"12986664-524e-49c5-af92-2d10265930ea\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") "
Jul 01 22:00:40.302162 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.302041    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "12986664-524e-49c5-af92-2d10265930ea" (UID: "12986664-524e-49c5-af92-2d10265930ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:40.302162 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.302064    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-home\") pod \"12986664-524e-49c5-af92-2d10265930ea\" (UID: \"12986664-524e-49c5-af92-2d10265930ea\") "
Jul 01 22:00:40.302379 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.302274    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12986664-524e-49c5-af92-2d10265930ea" (UID: "12986664-524e-49c5-af92-2d10265930ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:40.302379 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.302326    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:40.302497 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.302432    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-home" (OuterVolumeSpecName: "home") pod "12986664-524e-49c5-af92-2d10265930ea" (UID: "12986664-524e-49c5-af92-2d10265930ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:40.304111 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.304083    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12986664-524e-49c5-af92-2d10265930ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12986664-524e-49c5-af92-2d10265930ea" (UID: "12986664-524e-49c5-af92-2d10265930ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 22:00:40.304111 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.304092    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12986664-524e-49c5-af92-2d10265930ea-kube-api-access-7n862" (OuterVolumeSpecName: "kube-api-access-7n862") pod "12986664-524e-49c5-af92-2d10265930ea" (UID: "12986664-524e-49c5-af92-2d10265930ea"). InnerVolumeSpecName "kube-api-access-7n862". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:00:40.304368 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.304345    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-dshm" (OuterVolumeSpecName: "dshm") pod "12986664-524e-49c5-af92-2d10265930ea" (UID: "12986664-524e-49c5-af92-2d10265930ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:40.361861 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.361830    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12986664-524e-49c5-af92-2d10265930ea" (UID: "12986664-524e-49c5-af92-2d10265930ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:00:40.402896 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.402873    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n862\" (UniqueName: \"kubernetes.io/projected/12986664-524e-49c5-af92-2d10265930ea-kube-api-access-7n862\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:40.402896 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.402895    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-kserve-provision-location\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:40.403021 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.402905    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:40.403021 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.402914    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:40.403021 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.402922    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12986664-524e-49c5-af92-2d10265930ea-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:40.403021 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.402950    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12986664-524e-49c5-af92-2d10265930ea-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:00:40.992584 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.992556    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-6769cdd744-kpvrw_12986664-524e-49c5-af92-2d10265930ea/main/0.log"
Jul 01 22:00:40.992934 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.992912    2573 generic.go:356] "Generic (PLEG): container finished" podID="12986664-524e-49c5-af92-2d10265930ea" containerID="61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293" exitCode=130
Jul 01 22:00:40.993009 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.992981    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"
Jul 01 22:00:40.993054 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.993001    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" event={"ID":"12986664-524e-49c5-af92-2d10265930ea","Type":"ContainerDied","Data":"61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293"}
Jul 01 22:00:40.993092 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.993063    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw" event={"ID":"12986664-524e-49c5-af92-2d10265930ea","Type":"ContainerDied","Data":"f8b537cb43adb22b8c4897611dff6f2af96a6955d1831c9a525b084659d665c4"}
Jul 01 22:00:40.993130 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:40.993095    2573 scope.go:117] "RemoveContainer" containerID="61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293"
Jul 01 22:00:41.002897 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:41.002880    2573 scope.go:117] "RemoveContainer" containerID="b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c"
Jul 01 22:00:41.011249 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:41.011232    2573 scope.go:117] "RemoveContainer" containerID="61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293"
Jul 01 22:00:41.011490 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:00:41.011474    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293\": container with ID starting with 61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293 not found: ID does not exist" containerID="61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293"
Jul 01 22:00:41.011531 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:41.011498    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293"} err="failed to get container status \"61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293\": rpc error: code = NotFound desc = could not find container \"61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293\": container with ID starting with 61fd731703ddde9fe132139f15ed0c864612dee9cc7a1eb8cc2a43281bc78293 not found: ID does not exist"
Jul 01 22:00:41.011531 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:41.011516    2573 scope.go:117] "RemoveContainer" containerID="b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c"
Jul 01 22:00:41.011736 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:00:41.011714    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c\": container with ID starting with b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c not found: ID does not exist" containerID="b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c"
Jul 01 22:00:41.011866 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:41.011743    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c"} err="failed to get container status \"b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c\": rpc error: code = NotFound desc = could not find container \"b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c\": container with ID starting with b929c603b53cb2b0a6fdcbe4108351942dd0a705b86b5ed8c56e098405b3cc4c not found: ID does not exist"
Jul 01 22:00:41.031179 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:41.031153    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"]
Jul 01 22:00:41.037336 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:41.037317    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw"]
Jul 01 22:00:41.591300 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:41.591267    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12986664-524e-49c5-af92-2d10265930ea" path="/var/lib/kubelet/pods/12986664-524e-49c5-af92-2d10265930ea/volumes"
Jul 01 22:00:51.569846 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.569814    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"]
Jul 01 22:00:51.570258 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570157    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="storage-initializer"
Jul 01 22:00:51.570258 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570170    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="storage-initializer"
Jul 01 22:00:51.570258 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570187    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12986664-524e-49c5-af92-2d10265930ea" containerName="storage-initializer"
Jul 01 22:00:51.570258 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570192    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="12986664-524e-49c5-af92-2d10265930ea" containerName="storage-initializer"
Jul 01 22:00:51.570258 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570203    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main"
Jul 01 22:00:51.570258 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570208    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main"
Jul 01 22:00:51.570258 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570222    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12986664-524e-49c5-af92-2d10265930ea" containerName="main"
Jul 01 22:00:51.570258 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570230    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="12986664-524e-49c5-af92-2d10265930ea" containerName="main"
Jul 01 22:00:51.570505 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570280    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dfe85a7-7665-4391-adf3-ebd19a2a0c48" containerName="main"
Jul 01 22:00:51.570505 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.570289    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="12986664-524e-49c5-af92-2d10265930ea" containerName="main"
Jul 01 22:00:51.573000 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.572983    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.575619 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.575596    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 01 22:00:51.583608 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.583582    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"]
Jul 01 22:00:51.689789 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.689765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.689977 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.689796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.689977 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.689819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.689977 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.689907    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.690135 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.689973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mrvv\" (UniqueName: \"kubernetes.io/projected/905ec60a-60af-4558-b511-528d66f85cf0-kube-api-access-4mrvv\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.690135 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.690013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905ec60a-60af-4558-b511-528d66f85cf0-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.790686 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.790650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mrvv\" (UniqueName: \"kubernetes.io/projected/905ec60a-60af-4558-b511-528d66f85cf0-kube-api-access-4mrvv\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.790883 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.790703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905ec60a-60af-4558-b511-528d66f85cf0-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.790883 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.790823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.790883 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.790881    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.791093 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.790921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.791093 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.790971    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.791250 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.791228    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.791302 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.791274    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.791354 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.791312    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.793102 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.793078    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.793262 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.793246    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905ec60a-60af-4558-b511-528d66f85cf0-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.799508 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.799486    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mrvv\" (UniqueName: \"kubernetes.io/projected/905ec60a-60af-4558-b511-528d66f85cf0-kube-api-access-4mrvv\") pod \"rolling-upgrade-test-kserve-54bc76f55b-r2wjc\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:51.883037 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:51.882984    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:00:52.014344 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:52.014315    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"]
Jul 01 22:00:52.017101 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:00:52.017067    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905ec60a_60af_4558_b511_528d66f85cf0.slice/crio-54507b533fcbc7dba52dc4a43b0890e45b9723c7edbd0b70ec8e084a2f8dc22c WatchSource:0}: Error finding container 54507b533fcbc7dba52dc4a43b0890e45b9723c7edbd0b70ec8e084a2f8dc22c: Status 404 returned error can't find the container with id 54507b533fcbc7dba52dc4a43b0890e45b9723c7edbd0b70ec8e084a2f8dc22c
Jul 01 22:00:52.042148 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:52.042120    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc" event={"ID":"905ec60a-60af-4558-b511-528d66f85cf0","Type":"ContainerStarted","Data":"54507b533fcbc7dba52dc4a43b0890e45b9723c7edbd0b70ec8e084a2f8dc22c"}
Jul 01 22:00:53.048109 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:53.048066    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc" event={"ID":"905ec60a-60af-4558-b511-528d66f85cf0","Type":"ContainerStarted","Data":"ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681"}
Jul 01 22:00:53.067092 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:53.067048    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc" podStartSLOduration=2.067036247 podStartE2EDuration="2.067036247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 22:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:00:53.066765701 +0000 UTC m=+4485.978100326" watchObservedRunningTime="2026-07-01 22:00:53.067036247 +0000 UTC m=+4485.978370915"
Jul 01 22:00:53.883247 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:00:53.883217    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:01:01.883704 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:01.883609    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:01:01.895970 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:01.895943    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:01:02.097186 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:02.097158    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:01:07.957200 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:07.957092    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:01:07.975983 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:07.964070    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:01:07.975983 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:07.964211    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:01:07.975983 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:07.971124    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:01:08.788214 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:08.788177    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"]
Jul 01 22:01:08.800019 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:01:08.799984    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 22:01:08.800019 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 22:01:08.800019 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 22:01:08.800019 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b" cmd=["/bin/sleep","15"]
Jul 01 22:01:08.800338 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:08.800041    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6" message="" err=<
Jul 01 22:01:08.800338 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 22:01:08.800338 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 22:01:08.800338 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 22:01:08.800338 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:01:08.800061    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 22:01:08.800338 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 22:01:08.800338 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 22:01:08.800338 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6" podUID="5dedf8a4-8117-469a-bd30-34c8a2173219" containerName="main" containerID="cri-o://5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b"
Jul 01 22:01:08.800338 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:08.800113    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6" podUID="5dedf8a4-8117-469a-bd30-34c8a2173219" containerName="main" containerID="cri-o://5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b" gracePeriod=60
Jul 01 22:01:09.043766 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.043711    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 22:01:09.116070 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.116038    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5dedf8a4-8117-469a-bd30-34c8a2173219-tokenizer-uds\") pod \"5dedf8a4-8117-469a-bd30-34c8a2173219\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") "
Jul 01 22:01:09.116237 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.116156    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6qgq\" (UniqueName: \"kubernetes.io/projected/5dedf8a4-8117-469a-bd30-34c8a2173219-kube-api-access-b6qgq\") pod \"5dedf8a4-8117-469a-bd30-34c8a2173219\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") "
Jul 01 22:01:09.116237 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.116192    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dedf8a4-8117-469a-bd30-34c8a2173219-tls-certs\") pod \"5dedf8a4-8117-469a-bd30-34c8a2173219\" (UID: \"5dedf8a4-8117-469a-bd30-34c8a2173219\") "
Jul 01 22:01:09.116331 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.116304    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dedf8a4-8117-469a-bd30-34c8a2173219-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5dedf8a4-8117-469a-bd30-34c8a2173219" (UID: "5dedf8a4-8117-469a-bd30-34c8a2173219"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:01:09.116436 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.116406    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5dedf8a4-8117-469a-bd30-34c8a2173219-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:01:09.117196 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.117167    2573 generic.go:356] "Generic (PLEG): container finished" podID="5dedf8a4-8117-469a-bd30-34c8a2173219" containerID="5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b" exitCode=0
Jul 01 22:01:09.117293 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.117227    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6" event={"ID":"5dedf8a4-8117-469a-bd30-34c8a2173219","Type":"ContainerDied","Data":"5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b"}
Jul 01 22:01:09.117293 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.117283    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6" event={"ID":"5dedf8a4-8117-469a-bd30-34c8a2173219","Type":"ContainerDied","Data":"835caef2341896505f8602651c066e1f2e0372215031d5bab12977cd111fd2d1"}
Jul 01 22:01:09.117380 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.117312    2573 scope.go:117] "RemoveContainer" containerID="5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b"
Jul 01 22:01:09.117380 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.117232    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"
Jul 01 22:01:09.118325 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.118299    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dedf8a4-8117-469a-bd30-34c8a2173219-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5dedf8a4-8117-469a-bd30-34c8a2173219" (UID: "5dedf8a4-8117-469a-bd30-34c8a2173219"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 22:01:09.118491 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.118474    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dedf8a4-8117-469a-bd30-34c8a2173219-kube-api-access-b6qgq" (OuterVolumeSpecName: "kube-api-access-b6qgq") pod "5dedf8a4-8117-469a-bd30-34c8a2173219" (UID: "5dedf8a4-8117-469a-bd30-34c8a2173219"). InnerVolumeSpecName "kube-api-access-b6qgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:01:09.133491 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.133473    2573 scope.go:117] "RemoveContainer" containerID="5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b"
Jul 01 22:01:09.133733 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:01:09.133716    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b\": container with ID starting with 5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b not found: ID does not exist" containerID="5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b"
Jul 01 22:01:09.133790 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.133743    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b"} err="failed to get container status \"5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b\": rpc error: code = NotFound desc = could not find container \"5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b\": container with ID starting with 5f17ff389392f9beeed14e3875d1ef55a2b3e671ff9341483f9ec6a672552d4b not found: ID does not exist"
Jul 01 22:01:09.217349 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.217324    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6qgq\" (UniqueName: \"kubernetes.io/projected/5dedf8a4-8117-469a-bd30-34c8a2173219-kube-api-access-b6qgq\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:01:09.217349 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.217345    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dedf8a4-8117-469a-bd30-34c8a2173219-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:01:09.438269 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.438240    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"]
Jul 01 22:01:09.441555 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.441531    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6"]
Jul 01 22:01:09.590334 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:09.590304    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dedf8a4-8117-469a-bd30-34c8a2173219" path="/var/lib/kubelet/pods/5dedf8a4-8117-469a-bd30-34c8a2173219/volumes"
Jul 01 22:01:36.401238 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:36.401207    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"]
Jul 01 22:01:49.852172 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.852139    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"]
Jul 01 22:01:49.852569 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.852472    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dedf8a4-8117-469a-bd30-34c8a2173219" containerName="main"
Jul 01 22:01:49.852569 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.852483    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dedf8a4-8117-469a-bd30-34c8a2173219" containerName="main"
Jul 01 22:01:49.852569 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.852548    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dedf8a4-8117-469a-bd30-34c8a2173219" containerName="main"
Jul 01 22:01:49.854654 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.854637    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:49.857423 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.857402    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 01 22:01:49.864632 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.864607    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"]
Jul 01 22:01:49.923798 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.923775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:49.923928 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.923810    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2t64\" (UniqueName: \"kubernetes.io/projected/6db14807-8682-4240-a3af-49ded81cbfa6-kube-api-access-r2t64\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:49.923928 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.923831    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:49.923928 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.923887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:49.923928 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.923927    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db14807-8682-4240-a3af-49ded81cbfa6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:49.924086 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:49.923978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.024488 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.024463    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.024594 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.024494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2t64\" (UniqueName: \"kubernetes.io/projected/6db14807-8682-4240-a3af-49ded81cbfa6-kube-api-access-r2t64\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.024594 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.024515    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.024594 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.024546    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.024594 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.024579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db14807-8682-4240-a3af-49ded81cbfa6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.024810 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.024637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.024932 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.024912    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.024991 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.024960    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.024991 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.024981    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.026936 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.026917    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.027322 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.027300    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db14807-8682-4240-a3af-49ded81cbfa6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.033194 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.033172    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2t64\" (UniqueName: \"kubernetes.io/projected/6db14807-8682-4240-a3af-49ded81cbfa6-kube-api-access-r2t64\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.165213 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.165146    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:50.292285 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.292255    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"]
Jul 01 22:01:50.295625 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.295608    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.298900 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.298834    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-zbsdm\""
Jul 01 22:01:50.299359 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.299338    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"]
Jul 01 22:01:50.303563 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:01:50.303542    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db14807_8682_4240_a3af_49ded81cbfa6.slice/crio-72dc8973909386f01d0db1710912194afbd9b3bca1e590350c89365f8f92e5af WatchSource:0}: Error finding container 72dc8973909386f01d0db1710912194afbd9b3bca1e590350c89365f8f92e5af: Status 404 returned error can't find the container with id 72dc8973909386f01d0db1710912194afbd9b3bca1e590350c89365f8f92e5af
Jul 01 22:01:50.304301 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.304281    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"]
Jul 01 22:01:50.327637 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.327612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnbx\" (UniqueName: \"kubernetes.io/projected/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-kube-api-access-xbnbx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.327725 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.327661    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.327725 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.327685    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.428324 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.428263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnbx\" (UniqueName: \"kubernetes.io/projected/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-kube-api-access-xbnbx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.428324 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.428312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.428472 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.428334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.428661 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.428637    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.430548 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.430531    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.440356 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.440333    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnbx\" (UniqueName: \"kubernetes.io/projected/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-kube-api-access-xbnbx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.614906 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.614878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:50.737708 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:50.737578    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"]
Jul 01 22:01:50.740182 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:01:50.740148    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864ec2ad_10c9_49e3_9a7c_63f7aff822ba.slice/crio-536d4953f84054765895cc67f1f19a2e7ea8a013ad9e989704c52ca0054c81e7 WatchSource:0}: Error finding container 536d4953f84054765895cc67f1f19a2e7ea8a013ad9e989704c52ca0054c81e7: Status 404 returned error can't find the container with id 536d4953f84054765895cc67f1f19a2e7ea8a013ad9e989704c52ca0054c81e7
Jul 01 22:01:51.305816 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.305784    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" event={"ID":"864ec2ad-10c9-49e3-9a7c-63f7aff822ba","Type":"ContainerStarted","Data":"53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269"}
Jul 01 22:01:51.306271 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.305835    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" event={"ID":"864ec2ad-10c9-49e3-9a7c-63f7aff822ba","Type":"ContainerStarted","Data":"536d4953f84054765895cc67f1f19a2e7ea8a013ad9e989704c52ca0054c81e7"}
Jul 01 22:01:51.306271 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.305887    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:01:51.307322 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.307301    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b" event={"ID":"6db14807-8682-4240-a3af-49ded81cbfa6","Type":"ContainerStarted","Data":"a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0"}
Jul 01 22:01:51.307424 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.307340    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b" event={"ID":"6db14807-8682-4240-a3af-49ded81cbfa6","Type":"ContainerStarted","Data":"72dc8973909386f01d0db1710912194afbd9b3bca1e590350c89365f8f92e5af"}
Jul 01 22:01:51.323579 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.323537    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podStartSLOduration=1.323523303 podStartE2EDuration="1.323523303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 22:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:01:51.322557146 +0000 UTC m=+4544.233891773" watchObservedRunningTime="2026-07-01 22:01:51.323523303 +0000 UTC m=+4544.234857926"
Jul 01 22:01:51.338967 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.338929    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b" podStartSLOduration=2.338916967 podStartE2EDuration="2.338916967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 22:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:01:51.33801329 +0000 UTC m=+4544.249347918" watchObservedRunningTime="2026-07-01 22:01:51.338916967 +0000 UTC m=+4544.250251592"
Jul 01 22:01:51.426452 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.426423    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc" podUID="905ec60a-60af-4558-b511-528d66f85cf0" containerName="main" containerID="cri-o://ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681" gracePeriod=45
Jul 01 22:01:51.664192 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.664168    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:01:51.736588 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.736561    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905ec60a-60af-4558-b511-528d66f85cf0-tls-certs\") pod \"905ec60a-60af-4558-b511-528d66f85cf0\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") "
Jul 01 22:01:51.736696 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.736597    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-home\") pod \"905ec60a-60af-4558-b511-528d66f85cf0\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") "
Jul 01 22:01:51.736696 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.736619    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-model-cache\") pod \"905ec60a-60af-4558-b511-528d66f85cf0\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") "
Jul 01 22:01:51.736696 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.736634    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-tmp-dir\") pod \"905ec60a-60af-4558-b511-528d66f85cf0\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") "
Jul 01 22:01:51.736815 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.736768    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-dshm\") pod \"905ec60a-60af-4558-b511-528d66f85cf0\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") "
Jul 01 22:01:51.736914 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.736897    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mrvv\" (UniqueName: \"kubernetes.io/projected/905ec60a-60af-4558-b511-528d66f85cf0-kube-api-access-4mrvv\") pod \"905ec60a-60af-4558-b511-528d66f85cf0\" (UID: \"905ec60a-60af-4558-b511-528d66f85cf0\") "
Jul 01 22:01:51.737010 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.736898    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-home" (OuterVolumeSpecName: "home") pod "905ec60a-60af-4558-b511-528d66f85cf0" (UID: "905ec60a-60af-4558-b511-528d66f85cf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:01:51.737010 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.736911    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-model-cache" (OuterVolumeSpecName: "model-cache") pod "905ec60a-60af-4558-b511-528d66f85cf0" (UID: "905ec60a-60af-4558-b511-528d66f85cf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:01:51.737134 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.736923    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "905ec60a-60af-4558-b511-528d66f85cf0" (UID: "905ec60a-60af-4558-b511-528d66f85cf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:01:51.737203 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.737175    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:01:51.737203 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.737189    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:01:51.737203 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.737198    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:01:51.738799 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.738776    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-dshm" (OuterVolumeSpecName: "dshm") pod "905ec60a-60af-4558-b511-528d66f85cf0" (UID: "905ec60a-60af-4558-b511-528d66f85cf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:01:51.738870 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.738806    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905ec60a-60af-4558-b511-528d66f85cf0-kube-api-access-4mrvv" (OuterVolumeSpecName: "kube-api-access-4mrvv") pod "905ec60a-60af-4558-b511-528d66f85cf0" (UID: "905ec60a-60af-4558-b511-528d66f85cf0"). InnerVolumeSpecName "kube-api-access-4mrvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:01:51.738930 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.738911    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/905ec60a-60af-4558-b511-528d66f85cf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "905ec60a-60af-4558-b511-528d66f85cf0" (UID: "905ec60a-60af-4558-b511-528d66f85cf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 22:01:51.837964 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.837939    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905ec60a-60af-4558-b511-528d66f85cf0-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:01:51.837964 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.837960    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mrvv\" (UniqueName: \"kubernetes.io/projected/905ec60a-60af-4558-b511-528d66f85cf0-kube-api-access-4mrvv\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:01:51.838105 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:51.837972    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905ec60a-60af-4558-b511-528d66f85cf0-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:01:52.165649 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.165618    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:01:52.312737 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.312705    2573 generic.go:356] "Generic (PLEG): container finished" podID="905ec60a-60af-4558-b511-528d66f85cf0" containerID="ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681" exitCode=0
Jul 01 22:01:52.313220 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.312747    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"
Jul 01 22:01:52.313220 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.312796    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc" event={"ID":"905ec60a-60af-4558-b511-528d66f85cf0","Type":"ContainerDied","Data":"ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681"}
Jul 01 22:01:52.313220 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.312872    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc" event={"ID":"905ec60a-60af-4558-b511-528d66f85cf0","Type":"ContainerDied","Data":"54507b533fcbc7dba52dc4a43b0890e45b9723c7edbd0b70ec8e084a2f8dc22c"}
Jul 01 22:01:52.313220 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.312912    2573 scope.go:117] "RemoveContainer" containerID="ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681"
Jul 01 22:01:52.321763 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.321749    2573 scope.go:117] "RemoveContainer" containerID="ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681"
Jul 01 22:01:52.322032 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:01:52.322014    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681\": container with ID starting with ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681 not found: ID does not exist" containerID="ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681"
Jul 01 22:01:52.322090 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.322044    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681"} err="failed to get container status \"ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681\": rpc error: code = NotFound desc = could not find container \"ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681\": container with ID starting with ccbc3c07f303c6ecccce9cf2d79e79a0edb94f6fd4f4934ba2f8fa4053e35681 not found: ID does not exist"
Jul 01 22:01:52.334218 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.334194    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"]
Jul 01 22:01:52.340150 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:52.340131    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc"]
Jul 01 22:01:53.590628 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:01:53.590598    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="905ec60a-60af-4558-b511-528d66f85cf0" path="/var/lib/kubelet/pods/905ec60a-60af-4558-b511-528d66f85cf0/volumes"
Jul 01 22:02:00.165313 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:02:00.165278    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:02:00.177983 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:02:00.177955    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:02:00.360798 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:02:00.360766    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:02:22.314875 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:02:22.314814    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 22:02:32.314567 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:02:32.314466    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 22:02:42.314664 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:02:42.314623    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 22:02:52.314757 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:02:52.314715    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 22:03:02.314390 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:02.314349    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 22:03:02.586823 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:02.586783    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 22:03:12.586941 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:12.586902    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 22:03:22.587608 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:22.587560    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 22:03:26.625346 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.625306    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"]
Jul 01 22:03:26.625805 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.625629    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="905ec60a-60af-4558-b511-528d66f85cf0" containerName="main"
Jul 01 22:03:26.625805 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.625640    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="905ec60a-60af-4558-b511-528d66f85cf0" containerName="main"
Jul 01 22:03:26.625805 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.625697    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="905ec60a-60af-4558-b511-528d66f85cf0" containerName="main"
Jul 01 22:03:26.627496 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.627476    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.632440 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.632420    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-24ndg\""
Jul 01 22:03:26.632533 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.632421    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 01 22:03:26.640901 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.640880    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"]
Jul 01 22:03:26.764438 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.764403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1874861-4e64-470f-8964-44c6e8fd5683-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k\" (UID: \"b1874861-4e64-470f-8964-44c6e8fd5683\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.764587 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.764473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b1874861-4e64-470f-8964-44c6e8fd5683-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k\" (UID: \"b1874861-4e64-470f-8964-44c6e8fd5683\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.764587 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.764528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcdsh\" (UniqueName: \"kubernetes.io/projected/b1874861-4e64-470f-8964-44c6e8fd5683-kube-api-access-gcdsh\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k\" (UID: \"b1874861-4e64-470f-8964-44c6e8fd5683\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.864943 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.864916    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b1874861-4e64-470f-8964-44c6e8fd5683-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k\" (UID: \"b1874861-4e64-470f-8964-44c6e8fd5683\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.865085 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.864962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcdsh\" (UniqueName: \"kubernetes.io/projected/b1874861-4e64-470f-8964-44c6e8fd5683-kube-api-access-gcdsh\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k\" (UID: \"b1874861-4e64-470f-8964-44c6e8fd5683\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.865085 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.865007    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1874861-4e64-470f-8964-44c6e8fd5683-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k\" (UID: \"b1874861-4e64-470f-8964-44c6e8fd5683\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.865347 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.865324    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b1874861-4e64-470f-8964-44c6e8fd5683-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k\" (UID: \"b1874861-4e64-470f-8964-44c6e8fd5683\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.867526 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.867503    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1874861-4e64-470f-8964-44c6e8fd5683-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k\" (UID: \"b1874861-4e64-470f-8964-44c6e8fd5683\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.874535 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.874505    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcdsh\" (UniqueName: \"kubernetes.io/projected/b1874861-4e64-470f-8964-44c6e8fd5683-kube-api-access-gcdsh\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k\" (UID: \"b1874861-4e64-470f-8964-44c6e8fd5683\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:26.936734 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:26.936676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:27.067062 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:27.067038    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"]
Jul 01 22:03:27.069242 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:03:27.069210    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1874861_4e64_470f_8964_44c6e8fd5683.slice/crio-05a2eb1490b61740c01c8457696c9cad00dd7f97231604559096f67d07985eeb WatchSource:0}: Error finding container 05a2eb1490b61740c01c8457696c9cad00dd7f97231604559096f67d07985eeb: Status 404 returned error can't find the container with id 05a2eb1490b61740c01c8457696c9cad00dd7f97231604559096f67d07985eeb
Jul 01 22:03:27.730055 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:27.730017    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k" event={"ID":"b1874861-4e64-470f-8964-44c6e8fd5683","Type":"ContainerStarted","Data":"3fc52102dbec9138800878784ebdab0d26834bc02750b36350ac9e89cef8960c"}
Jul 01 22:03:27.730479 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:27.730066    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k" event={"ID":"b1874861-4e64-470f-8964-44c6e8fd5683","Type":"ContainerStarted","Data":"05a2eb1490b61740c01c8457696c9cad00dd7f97231604559096f67d07985eeb"}
Jul 01 22:03:27.730479 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:27.730176    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:03:27.746072 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:27.746027    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k" podStartSLOduration=1.7460127239999998 podStartE2EDuration="1.746012724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 22:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:03:27.745367904 +0000 UTC m=+4640.656702555" watchObservedRunningTime="2026-07-01 22:03:27.746012724 +0000 UTC m=+4640.657347352"
Jul 01 22:03:29.682129 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.682094    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"]
Jul 01 22:03:29.683755 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.683720    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 01 22:03:29.686641 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.686617    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"]
Jul 01 22:03:29.692819 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:03:29.692783    2573 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 01 22:03:29.692819 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 22:03:29.692819 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 22:03:29.692819 ip-10-0-138-219 kubenswrapper[2573]:  > containerID="53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269" cmd=["/bin/sleep","15"]
Jul 01 22:03:29.693035 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.692842    2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" message="" err=<
Jul 01 22:03:29.693035 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 22:03:29.693035 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 22:03:29.693035 ip-10-0-138-219 kubenswrapper[2573]:  >
Jul 01 22:03:29.693035 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:03:29.692871    2573 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 01 22:03:29.693035 ip-10-0-138-219 kubenswrapper[2573]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 01 22:03:29.693035 ip-10-0-138-219 kubenswrapper[2573]:         , stdout: , stderr: , exit code -1
Jul 01 22:03:29.693035 ip-10-0-138-219 kubenswrapper[2573]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" containerID="cri-o://53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269"
Jul 01 22:03:29.693035 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.692912    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main" containerID="cri-o://53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269" gracePeriod=60
Jul 01 22:03:29.933880 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.933811    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:03:29.987805 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.987782    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tls-certs\") pod \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") "
Jul 01 22:03:29.987950 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.987882    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tokenizer-uds\") pod \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") "
Jul 01 22:03:29.987950 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.987926    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbnbx\" (UniqueName: \"kubernetes.io/projected/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-kube-api-access-xbnbx\") pod \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\" (UID: \"864ec2ad-10c9-49e3-9a7c-63f7aff822ba\") "
Jul 01 22:03:29.988147 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.988125    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "864ec2ad-10c9-49e3-9a7c-63f7aff822ba" (UID: "864ec2ad-10c9-49e3-9a7c-63f7aff822ba"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:03:29.989744 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.989721    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "864ec2ad-10c9-49e3-9a7c-63f7aff822ba" (UID: "864ec2ad-10c9-49e3-9a7c-63f7aff822ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 22:03:29.989868 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:29.989836    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-kube-api-access-xbnbx" (OuterVolumeSpecName: "kube-api-access-xbnbx") pod "864ec2ad-10c9-49e3-9a7c-63f7aff822ba" (UID: "864ec2ad-10c9-49e3-9a7c-63f7aff822ba"). InnerVolumeSpecName "kube-api-access-xbnbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:03:30.089000 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.088976    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbnbx\" (UniqueName: \"kubernetes.io/projected/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-kube-api-access-xbnbx\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:03:30.089000 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.088997    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:03:30.089128 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.089009    2573 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/864ec2ad-10c9-49e3-9a7c-63f7aff822ba-tokenizer-uds\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:03:30.745268 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.745238    2573 generic.go:356] "Generic (PLEG): container finished" podID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerID="53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269" exitCode=0
Jul 01 22:03:30.745677 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.745282    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" event={"ID":"864ec2ad-10c9-49e3-9a7c-63f7aff822ba","Type":"ContainerDied","Data":"53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269"}
Jul 01 22:03:30.745677 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.745315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9" event={"ID":"864ec2ad-10c9-49e3-9a7c-63f7aff822ba","Type":"ContainerDied","Data":"536d4953f84054765895cc67f1f19a2e7ea8a013ad9e989704c52ca0054c81e7"}
Jul 01 22:03:30.745677 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.745333    2573 scope.go:117] "RemoveContainer" containerID="53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269"
Jul 01 22:03:30.745677 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.745284    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"
Jul 01 22:03:30.755152 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.755135    2573 scope.go:117] "RemoveContainer" containerID="53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269"
Jul 01 22:03:30.755380 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:03:30.755358    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269\": container with ID starting with 53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269 not found: ID does not exist" containerID="53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269"
Jul 01 22:03:30.755432 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.755386    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269"} err="failed to get container status \"53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269\": rpc error: code = NotFound desc = could not find container \"53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269\": container with ID starting with 53035fe21bacd5ad8cf409e665831da2de3905cf69832601c8cd8a8be99b1269 not found: ID does not exist"
Jul 01 22:03:30.767298 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.767274    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"]
Jul 01 22:03:30.771552 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:30.771529    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9"]
Jul 01 22:03:31.596874 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:31.596833    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" path="/var/lib/kubelet/pods/864ec2ad-10c9-49e3-9a7c-63f7aff822ba/volumes"
Jul 01 22:03:44.712560 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:44.712499    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b" podUID="6db14807-8682-4240-a3af-49ded81cbfa6" containerName="main" containerID="cri-o://a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0" gracePeriod=45
Jul 01 22:03:44.969273 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:44.969211    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:03:45.106895 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.106867    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-model-cache\") pod \"6db14807-8682-4240-a3af-49ded81cbfa6\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") "
Jul 01 22:03:45.107091 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.106929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-home\") pod \"6db14807-8682-4240-a3af-49ded81cbfa6\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") "
Jul 01 22:03:45.107091 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.106961    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-dshm\") pod \"6db14807-8682-4240-a3af-49ded81cbfa6\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") "
Jul 01 22:03:45.107091 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.106985    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db14807-8682-4240-a3af-49ded81cbfa6-tls-certs\") pod \"6db14807-8682-4240-a3af-49ded81cbfa6\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") "
Jul 01 22:03:45.107091 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.107004    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-tmp-dir\") pod \"6db14807-8682-4240-a3af-49ded81cbfa6\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") "
Jul 01 22:03:45.107091 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.107047    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2t64\" (UniqueName: \"kubernetes.io/projected/6db14807-8682-4240-a3af-49ded81cbfa6-kube-api-access-r2t64\") pod \"6db14807-8682-4240-a3af-49ded81cbfa6\" (UID: \"6db14807-8682-4240-a3af-49ded81cbfa6\") "
Jul 01 22:03:45.107352 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.107122    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-model-cache" (OuterVolumeSpecName: "model-cache") pod "6db14807-8682-4240-a3af-49ded81cbfa6" (UID: "6db14807-8682-4240-a3af-49ded81cbfa6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:03:45.107352 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.107237    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:03:45.107352 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.107241    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-home" (OuterVolumeSpecName: "home") pod "6db14807-8682-4240-a3af-49ded81cbfa6" (UID: "6db14807-8682-4240-a3af-49ded81cbfa6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:03:45.107352 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.107324    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6db14807-8682-4240-a3af-49ded81cbfa6" (UID: "6db14807-8682-4240-a3af-49ded81cbfa6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:03:45.109131 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.109109    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db14807-8682-4240-a3af-49ded81cbfa6-kube-api-access-r2t64" (OuterVolumeSpecName: "kube-api-access-r2t64") pod "6db14807-8682-4240-a3af-49ded81cbfa6" (UID: "6db14807-8682-4240-a3af-49ded81cbfa6"). InnerVolumeSpecName "kube-api-access-r2t64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:03:45.109213 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.109131    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-dshm" (OuterVolumeSpecName: "dshm") pod "6db14807-8682-4240-a3af-49ded81cbfa6" (UID: "6db14807-8682-4240-a3af-49ded81cbfa6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:03:45.109252 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.109214    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db14807-8682-4240-a3af-49ded81cbfa6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6db14807-8682-4240-a3af-49ded81cbfa6" (UID: "6db14807-8682-4240-a3af-49ded81cbfa6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 22:03:45.207911 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.207887    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2t64\" (UniqueName: \"kubernetes.io/projected/6db14807-8682-4240-a3af-49ded81cbfa6-kube-api-access-r2t64\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:03:45.207911 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.207908    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:03:45.208062 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.207918    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:03:45.208062 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.207927    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db14807-8682-4240-a3af-49ded81cbfa6-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:03:45.208062 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.207935    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db14807-8682-4240-a3af-49ded81cbfa6-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:03:45.812605 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.812570    2573 generic.go:356] "Generic (PLEG): container finished" podID="6db14807-8682-4240-a3af-49ded81cbfa6" containerID="a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0" exitCode=0
Jul 01 22:03:45.813051 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.812624    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"
Jul 01 22:03:45.813051 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.812624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b" event={"ID":"6db14807-8682-4240-a3af-49ded81cbfa6","Type":"ContainerDied","Data":"a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0"}
Jul 01 22:03:45.813051 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.812767    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b" event={"ID":"6db14807-8682-4240-a3af-49ded81cbfa6","Type":"ContainerDied","Data":"72dc8973909386f01d0db1710912194afbd9b3bca1e590350c89365f8f92e5af"}
Jul 01 22:03:45.813051 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.812795    2573 scope.go:117] "RemoveContainer" containerID="a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0"
Jul 01 22:03:45.822250 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.822234    2573 scope.go:117] "RemoveContainer" containerID="a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0"
Jul 01 22:03:45.822523 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:03:45.822506    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0\": container with ID starting with a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0 not found: ID does not exist" containerID="a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0"
Jul 01 22:03:45.822571 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.822531    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0"} err="failed to get container status \"a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0\": rpc error: code = NotFound desc = could not find container \"a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0\": container with ID starting with a1657b2b8ffbb9984b5354072c4bd80cffebfa1873443dc4a556646fa6f607d0 not found: ID does not exist"
Jul 01 22:03:45.828742 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.828721    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"]
Jul 01 22:03:45.831211 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:45.831188    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b"]
Jul 01 22:03:47.590927 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:47.590894    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db14807-8682-4240-a3af-49ded81cbfa6" path="/var/lib/kubelet/pods/6db14807-8682-4240-a3af-49ded81cbfa6/volumes"
Jul 01 22:03:58.736669 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:03:58.736641    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k"
Jul 01 22:04:03.172479 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.172400    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"]
Jul 01 22:04:03.172817 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.172695    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db14807-8682-4240-a3af-49ded81cbfa6" containerName="main"
Jul 01 22:04:03.172817 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.172705    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db14807-8682-4240-a3af-49ded81cbfa6" containerName="main"
Jul 01 22:04:03.172817 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.172711    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main"
Jul 01 22:04:03.172817 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.172717    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main"
Jul 01 22:04:03.172817 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.172776    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db14807-8682-4240-a3af-49ded81cbfa6" containerName="main"
Jul 01 22:04:03.172817 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.172787    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="864ec2ad-10c9-49e3-9a7c-63f7aff822ba" containerName="main"
Jul 01 22:04:03.175465 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.175450    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.178511 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.178494    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 01 22:04:03.184481 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.184459    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"]
Jul 01 22:04:03.343083 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.343056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.343242 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.343090    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.343242 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.343128    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.343242 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.343178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x48xs\" (UniqueName: \"kubernetes.io/projected/3d627f65-3023-4c9a-ad73-2c0979a06280-kube-api-access-x48xs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.343242 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.343233    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.343385 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.343252    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d627f65-3023-4c9a-ad73-2c0979a06280-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.443871 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.443780    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.443871 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.443824    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.444094 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.443912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.444094 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.443935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x48xs\" (UniqueName: \"kubernetes.io/projected/3d627f65-3023-4c9a-ad73-2c0979a06280-kube-api-access-x48xs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.444094 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.443960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.444094 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.444033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d627f65-3023-4c9a-ad73-2c0979a06280-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.444355 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.444193    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.444355 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.444226    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.444450 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.444389    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.446226 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.446203    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.446528 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.446511    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d627f65-3023-4c9a-ad73-2c0979a06280-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.452496 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.452476    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x48xs\" (UniqueName: \"kubernetes.io/projected/3d627f65-3023-4c9a-ad73-2c0979a06280-kube-api-access-x48xs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.486167 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.486143    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:03.615222 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.615163    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"]
Jul 01 22:04:03.617614 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:04:03.617585    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d627f65_3023_4c9a_ad73_2c0979a06280.slice/crio-3860c073c34661b4615bb564f77d978b39a48f507d2b938048ca348a19f454dd WatchSource:0}: Error finding container 3860c073c34661b4615bb564f77d978b39a48f507d2b938048ca348a19f454dd: Status 404 returned error can't find the container with id 3860c073c34661b4615bb564f77d978b39a48f507d2b938048ca348a19f454dd
Jul 01 22:04:03.895259 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.895219    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9" event={"ID":"3d627f65-3023-4c9a-ad73-2c0979a06280","Type":"ContainerStarted","Data":"5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48"}
Jul 01 22:04:03.895444 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.895286    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9" event={"ID":"3d627f65-3023-4c9a-ad73-2c0979a06280","Type":"ContainerStarted","Data":"3860c073c34661b4615bb564f77d978b39a48f507d2b938048ca348a19f454dd"}
Jul 01 22:04:03.913536 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:03.913492    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9" podStartSLOduration=0.913479633 podStartE2EDuration="913.479633ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 22:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:04:03.911778631 +0000 UTC m=+4676.823113257" watchObservedRunningTime="2026-07-01 22:04:03.913479633 +0000 UTC m=+4676.824814258"
Jul 01 22:04:04.486517 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:04.486488    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:13.486457 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:13.486426    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:13.499357 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:13.499329    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:13.952691 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:13.952655    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:36.403445 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:36.403410    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"]
Jul 01 22:04:51.423800 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.423757    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9" podUID="3d627f65-3023-4c9a-ad73-2c0979a06280" containerName="main" containerID="cri-o://5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48" gracePeriod=45
Jul 01 22:04:51.677253 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.677194    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:51.811835 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.811806    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-dshm\") pod \"3d627f65-3023-4c9a-ad73-2c0979a06280\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") "
Jul 01 22:04:51.812041 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.811848    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-home\") pod \"3d627f65-3023-4c9a-ad73-2c0979a06280\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") "
Jul 01 22:04:51.812041 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.811894    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-tmp-dir\") pod \"3d627f65-3023-4c9a-ad73-2c0979a06280\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") "
Jul 01 22:04:51.812041 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.811923    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d627f65-3023-4c9a-ad73-2c0979a06280-tls-certs\") pod \"3d627f65-3023-4c9a-ad73-2c0979a06280\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") "
Jul 01 22:04:51.812041 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.811950    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x48xs\" (UniqueName: \"kubernetes.io/projected/3d627f65-3023-4c9a-ad73-2c0979a06280-kube-api-access-x48xs\") pod \"3d627f65-3023-4c9a-ad73-2c0979a06280\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") "
Jul 01 22:04:51.812041 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.811976    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-model-cache\") pod \"3d627f65-3023-4c9a-ad73-2c0979a06280\" (UID: \"3d627f65-3023-4c9a-ad73-2c0979a06280\") "
Jul 01 22:04:51.812337 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.812109    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-home" (OuterVolumeSpecName: "home") pod "3d627f65-3023-4c9a-ad73-2c0979a06280" (UID: "3d627f65-3023-4c9a-ad73-2c0979a06280"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:04:51.812337 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.812208    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:04:51.812337 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.812200    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d627f65-3023-4c9a-ad73-2c0979a06280" (UID: "3d627f65-3023-4c9a-ad73-2c0979a06280"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:04:51.812337 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.812269    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d627f65-3023-4c9a-ad73-2c0979a06280" (UID: "3d627f65-3023-4c9a-ad73-2c0979a06280"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:04:51.814046 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.814023    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-dshm" (OuterVolumeSpecName: "dshm") pod "3d627f65-3023-4c9a-ad73-2c0979a06280" (UID: "3d627f65-3023-4c9a-ad73-2c0979a06280"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:04:51.814168 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.814063    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d627f65-3023-4c9a-ad73-2c0979a06280-kube-api-access-x48xs" (OuterVolumeSpecName: "kube-api-access-x48xs") pod "3d627f65-3023-4c9a-ad73-2c0979a06280" (UID: "3d627f65-3023-4c9a-ad73-2c0979a06280"). InnerVolumeSpecName "kube-api-access-x48xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:04:51.814168 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.814064    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d627f65-3023-4c9a-ad73-2c0979a06280-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d627f65-3023-4c9a-ad73-2c0979a06280" (UID: "3d627f65-3023-4c9a-ad73-2c0979a06280"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 22:04:51.912678 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.912657    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:04:51.912802 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.912680    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:04:51.912802 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.912694    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d627f65-3023-4c9a-ad73-2c0979a06280-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:04:51.912802 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.912706    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x48xs\" (UniqueName: \"kubernetes.io/projected/3d627f65-3023-4c9a-ad73-2c0979a06280-kube-api-access-x48xs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:04:51.912802 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:51.912719    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d627f65-3023-4c9a-ad73-2c0979a06280-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:04:52.115315 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:52.115290    2573 generic.go:356] "Generic (PLEG): container finished" podID="3d627f65-3023-4c9a-ad73-2c0979a06280" containerID="5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48" exitCode=0
Jul 01 22:04:52.115456 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:52.115335    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"
Jul 01 22:04:52.115456 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:52.115370    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9" event={"ID":"3d627f65-3023-4c9a-ad73-2c0979a06280","Type":"ContainerDied","Data":"5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48"}
Jul 01 22:04:52.115456 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:52.115409    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9" event={"ID":"3d627f65-3023-4c9a-ad73-2c0979a06280","Type":"ContainerDied","Data":"3860c073c34661b4615bb564f77d978b39a48f507d2b938048ca348a19f454dd"}
Jul 01 22:04:52.115456 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:52.115431    2573 scope.go:117] "RemoveContainer" containerID="5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48"
Jul 01 22:04:52.124371 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:52.124355    2573 scope.go:117] "RemoveContainer" containerID="5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48"
Jul 01 22:04:52.124626 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:04:52.124609    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48\": container with ID starting with 5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48 not found: ID does not exist" containerID="5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48"
Jul 01 22:04:52.124678 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:52.124636    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48"} err="failed to get container status \"5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48\": rpc error: code = NotFound desc = could not find container \"5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48\": container with ID starting with 5225e9aa0f23e7db384ae31e1be8b487b361a1868193bf5241fed5e0aad72e48 not found: ID does not exist"
Jul 01 22:04:52.136892 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:52.136865    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"]
Jul 01 22:04:52.140803 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:52.140784    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9"]
Jul 01 22:04:53.591387 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:04:53.591356    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d627f65-3023-4c9a-ad73-2c0979a06280" path="/var/lib/kubelet/pods/3d627f65-3023-4c9a-ad73-2c0979a06280/volumes"
Jul 01 22:06:07.983926 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:06:07.983801    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:06:07.990497 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:06:07.990477    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:06:07.991791 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:06:07.991775    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:06:07.998066 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:06:07.998051    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:11:08.009751 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:11:08.009646    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:11:08.016673 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:11:08.016650    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:11:08.018969 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:11:08.018953    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:11:08.028203 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:11:08.028182    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:16:08.036342 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:16:08.036244    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:16:08.043308 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:16:08.043287    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:16:08.049152 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:16:08.049136    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:16:08.055545 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:16:08.055530    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:21:08.066116 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:21:08.066082    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:21:08.074020 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:21:08.073990    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:21:08.078645 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:21:08.078627    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:21:08.085555 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:21:08.085536    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:25:23.895970 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:23.895896    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:25:44.405431 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.405399    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"]
Jul 01 22:25:44.405811 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.405707    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d627f65-3023-4c9a-ad73-2c0979a06280" containerName="main"
Jul 01 22:25:44.405811 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.405718    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d627f65-3023-4c9a-ad73-2c0979a06280" containerName="main"
Jul 01 22:25:44.405811 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.405779    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d627f65-3023-4c9a-ad73-2c0979a06280" containerName="main"
Jul 01 22:25:44.408556 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.408536    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"]
Jul 01 22:25:44.408684 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.408664    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.411526 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.411506    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 01 22:25:44.412418 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.412397    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.414730 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.414711    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-wwrqh\""
Jul 01 22:25:44.427614 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.427582    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"]
Jul 01 22:25:44.429036 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.429016    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"]
Jul 01 22:25:44.493436 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493408    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b047642c-5aaf-4278-8f57-f7b56b602c3f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.493590 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493441    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flsd\" (UniqueName: \"kubernetes.io/projected/b047642c-5aaf-4278-8f57-f7b56b602c3f-kube-api-access-8flsd\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.493590 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.493590 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493478    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.493590 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493508    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.493590 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493522    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.493590 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svsrl\" (UniqueName: \"kubernetes.io/projected/61e3e504-acf4-421d-b7c5-69b7bc9359bb-kube-api-access-svsrl\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.493823 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493595    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.493823 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493620    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.493823 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493653    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.493823 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493673    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.493823 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.493695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.594819 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.594794    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b047642c-5aaf-4278-8f57-f7b56b602c3f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.594992 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.594821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flsd\" (UniqueName: \"kubernetes.io/projected/b047642c-5aaf-4278-8f57-f7b56b602c3f-kube-api-access-8flsd\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.594992 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.594842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.594992 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.594875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.595171 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.595171 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595081    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.595171 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svsrl\" (UniqueName: \"kubernetes.io/projected/61e3e504-acf4-421d-b7c5-69b7bc9359bb-kube-api-access-svsrl\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.595325 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595181    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.595325 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.595325 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595245    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.595325 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595274    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.595325 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595282    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.595325 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595296    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.595325 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.595677 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595332    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.595677 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595445    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.595677 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595572    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.595803 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.595783    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.597408 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.597383    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.597516 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.597391    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.597681 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.597664    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b047642c-5aaf-4278-8f57-f7b56b602c3f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.597778 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.597761    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.604165 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.604133    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flsd\" (UniqueName: \"kubernetes.io/projected/b047642c-5aaf-4278-8f57-f7b56b602c3f-kube-api-access-8flsd\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.604415 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.604398    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svsrl\" (UniqueName: \"kubernetes.io/projected/61e3e504-acf4-421d-b7c5-69b7bc9359bb-kube-api-access-svsrl\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.720658 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.720605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:44.725669 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.725654    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:44.861712 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.861682    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"]
Jul 01 22:25:44.865673 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:25:44.865645    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb047642c_5aaf_4278_8f57_f7b56b602c3f.slice/crio-038699b9d998810fd614eb57d52faf5f1914c265b1d8250864a2be3441b73491 WatchSource:0}: Error finding container 038699b9d998810fd614eb57d52faf5f1914c265b1d8250864a2be3441b73491: Status 404 returned error can't find the container with id 038699b9d998810fd614eb57d52faf5f1914c265b1d8250864a2be3441b73491
Jul 01 22:25:44.885787 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:44.885070    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"]
Jul 01 22:25:44.896028 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:25:44.895998    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e3e504_acf4_421d_b7c5_69b7bc9359bb.slice/crio-83bad797dcef8c26d27229006cd7a8e27920ad7ed0142affd6664953d67b3484 WatchSource:0}: Error finding container 83bad797dcef8c26d27229006cd7a8e27920ad7ed0142affd6664953d67b3484: Status 404 returned error can't find the container with id 83bad797dcef8c26d27229006cd7a8e27920ad7ed0142affd6664953d67b3484
Jul 01 22:25:45.531632 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:45.531587    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr" event={"ID":"b047642c-5aaf-4278-8f57-f7b56b602c3f","Type":"ContainerStarted","Data":"a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3"}
Jul 01 22:25:45.532147 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:45.531664    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:45.532147 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:45.531691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr" event={"ID":"b047642c-5aaf-4278-8f57-f7b56b602c3f","Type":"ContainerStarted","Data":"038699b9d998810fd614eb57d52faf5f1914c265b1d8250864a2be3441b73491"}
Jul 01 22:25:45.533474 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:45.533442    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm" event={"ID":"61e3e504-acf4-421d-b7c5-69b7bc9359bb","Type":"ContainerStarted","Data":"9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186"}
Jul 01 22:25:45.533614 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:45.533501    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm" event={"ID":"61e3e504-acf4-421d-b7c5-69b7bc9359bb","Type":"ContainerStarted","Data":"83bad797dcef8c26d27229006cd7a8e27920ad7ed0142affd6664953d67b3484"}
Jul 01 22:25:45.571781 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:45.571729    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm" podStartSLOduration=1.5717145430000001 podStartE2EDuration="1.571714543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 22:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:25:45.569355957 +0000 UTC m=+5978.480690587" watchObservedRunningTime="2026-07-01 22:25:45.571714543 +0000 UTC m=+5978.483049169"
Jul 01 22:25:45.721221 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:45.721197    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:46.540231 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:46.540193    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr" event={"ID":"b047642c-5aaf-4278-8f57-f7b56b602c3f","Type":"ContainerStarted","Data":"04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313"}
Jul 01 22:25:46.562825 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:46.562779    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr" podStartSLOduration=2.562762233 podStartE2EDuration="2.562762233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 22:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:25:46.559988958 +0000 UTC m=+5979.471323600" watchObservedRunningTime="2026-07-01 22:25:46.562762233 +0000 UTC m=+5979.474096860"
Jul 01 22:25:46.726695 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:46.726659    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:54.721540 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:54.721465    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:54.725882 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:54.725846    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:54.737896 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:54.737869    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:54.738822 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:54.738796    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:54.742986 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:54.742966    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:25:55.601812 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:55.601780    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:25:55.602205 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:25:55.601938    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:26:08.094215 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:26:08.094072    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:26:08.100832 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:26:08.100814    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:26:08.105568 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:26:08.105551    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:26:08.111639 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:26:08.111623    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:31:08.119904 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:31:08.119784    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:31:08.127204 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:31:08.127182    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:31:08.132790 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:31:08.132773    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:31:08.139399 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:31:08.139384    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lrvqz_4c9ee537-12c1-4915-a6e4-46fe4784a8c4/ovn-acl-logging/0.log"
Jul 01 22:32:18.569684 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:18.569648    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"]
Jul 01 22:32:18.575403 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:18.575373    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"]
Jul 01 22:32:33.595993 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.595911    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm" podUID="61e3e504-acf4-421d-b7c5-69b7bc9359bb" containerName="main" containerID="cri-o://9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186" gracePeriod=45
Jul 01 22:32:33.602192 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.602143    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr" podUID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerName="main" containerID="cri-o://04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313" gracePeriod=45
Jul 01 22:32:33.712168 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.712134    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr" podUID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerName="llm-d-routing-sidecar" containerID="cri-o://a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3" gracePeriod=45
Jul 01 22:32:33.850091 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.850037    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:32:33.960538 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.960516    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:32:33.960660 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.960545    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tmp-dir\") pod \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") "
Jul 01 22:32:33.960660 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.960596    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svsrl\" (UniqueName: \"kubernetes.io/projected/61e3e504-acf4-421d-b7c5-69b7bc9359bb-kube-api-access-svsrl\") pod \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") "
Jul 01 22:32:33.960660 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.960637    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-dshm\") pod \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") "
Jul 01 22:32:33.960815 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.960670    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-home\") pod \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") "
Jul 01 22:32:33.960815 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.960688    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tls-certs\") pod \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") "
Jul 01 22:32:33.960815 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.960709    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-model-cache\") pod \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\" (UID: \"61e3e504-acf4-421d-b7c5-69b7bc9359bb\") "
Jul 01 22:32:33.961014 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.960843    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61e3e504-acf4-421d-b7c5-69b7bc9359bb" (UID: "61e3e504-acf4-421d-b7c5-69b7bc9359bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:32:33.961014 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.960936    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-home" (OuterVolumeSpecName: "home") pod "61e3e504-acf4-421d-b7c5-69b7bc9359bb" (UID: "61e3e504-acf4-421d-b7c5-69b7bc9359bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:32:33.961126 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.961082    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "61e3e504-acf4-421d-b7c5-69b7bc9359bb" (UID: "61e3e504-acf4-421d-b7c5-69b7bc9359bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:32:33.962665 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.962641    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e3e504-acf4-421d-b7c5-69b7bc9359bb-kube-api-access-svsrl" (OuterVolumeSpecName: "kube-api-access-svsrl") pod "61e3e504-acf4-421d-b7c5-69b7bc9359bb" (UID: "61e3e504-acf4-421d-b7c5-69b7bc9359bb"). InnerVolumeSpecName "kube-api-access-svsrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:32:33.962746 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.962705    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-dshm" (OuterVolumeSpecName: "dshm") pod "61e3e504-acf4-421d-b7c5-69b7bc9359bb" (UID: "61e3e504-acf4-421d-b7c5-69b7bc9359bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:32:33.962746 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:33.962708    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61e3e504-acf4-421d-b7c5-69b7bc9359bb" (UID: "61e3e504-acf4-421d-b7c5-69b7bc9359bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 22:32:34.061238 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061212    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-tmp-dir\") pod \"b047642c-5aaf-4278-8f57-f7b56b602c3f\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") "
Jul 01 22:32:34.061413 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061255    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-home\") pod \"b047642c-5aaf-4278-8f57-f7b56b602c3f\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") "
Jul 01 22:32:34.061413 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061277    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-model-cache\") pod \"b047642c-5aaf-4278-8f57-f7b56b602c3f\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") "
Jul 01 22:32:34.061413 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061318    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b047642c-5aaf-4278-8f57-f7b56b602c3f-tls-certs\") pod \"b047642c-5aaf-4278-8f57-f7b56b602c3f\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") "
Jul 01 22:32:34.061413 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061342    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-dshm\") pod \"b047642c-5aaf-4278-8f57-f7b56b602c3f\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") "
Jul 01 22:32:34.061413 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061384    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8flsd\" (UniqueName: \"kubernetes.io/projected/b047642c-5aaf-4278-8f57-f7b56b602c3f-kube-api-access-8flsd\") pod \"b047642c-5aaf-4278-8f57-f7b56b602c3f\" (UID: \"b047642c-5aaf-4278-8f57-f7b56b602c3f\") "
Jul 01 22:32:34.061668 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061460    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b047642c-5aaf-4278-8f57-f7b56b602c3f" (UID: "b047642c-5aaf-4278-8f57-f7b56b602c3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:32:34.061668 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061476    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-home" (OuterVolumeSpecName: "home") pod "b047642c-5aaf-4278-8f57-f7b56b602c3f" (UID: "b047642c-5aaf-4278-8f57-f7b56b602c3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:32:34.061668 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061571    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b047642c-5aaf-4278-8f57-f7b56b602c3f" (UID: "b047642c-5aaf-4278-8f57-f7b56b602c3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:32:34.061668 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061628    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.061668 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061649    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.061668 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061663    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.061933 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061677    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.061933 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061706    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.061933 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061716    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svsrl\" (UniqueName: \"kubernetes.io/projected/61e3e504-acf4-421d-b7c5-69b7bc9359bb-kube-api-access-svsrl\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.061933 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061725    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.061933 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.061734    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e3e504-acf4-421d-b7c5-69b7bc9359bb-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.063286 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.063261    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-dshm" (OuterVolumeSpecName: "dshm") pod "b047642c-5aaf-4278-8f57-f7b56b602c3f" (UID: "b047642c-5aaf-4278-8f57-f7b56b602c3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:32:34.063365 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.063347    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b047642c-5aaf-4278-8f57-f7b56b602c3f-kube-api-access-8flsd" (OuterVolumeSpecName: "kube-api-access-8flsd") pod "b047642c-5aaf-4278-8f57-f7b56b602c3f" (UID: "b047642c-5aaf-4278-8f57-f7b56b602c3f"). InnerVolumeSpecName "kube-api-access-8flsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:32:34.063411 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.063394    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b047642c-5aaf-4278-8f57-f7b56b602c3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b047642c-5aaf-4278-8f57-f7b56b602c3f" (UID: "b047642c-5aaf-4278-8f57-f7b56b602c3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 22:32:34.162699 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.162632    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.162699 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.162653    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b047642c-5aaf-4278-8f57-f7b56b602c3f-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.162699 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.162662    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b047642c-5aaf-4278-8f57-f7b56b602c3f-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.162699 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.162669    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8flsd\" (UniqueName: \"kubernetes.io/projected/b047642c-5aaf-4278-8f57-f7b56b602c3f-kube-api-access-8flsd\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:32:34.343933 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.343904    2573 generic.go:356] "Generic (PLEG): container finished" podID="61e3e504-acf4-421d-b7c5-69b7bc9359bb" containerID="9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186" exitCode=0
Jul 01 22:32:34.344103 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.343949    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"
Jul 01 22:32:34.344103 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.343989    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm" event={"ID":"61e3e504-acf4-421d-b7c5-69b7bc9359bb","Type":"ContainerDied","Data":"9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186"}
Jul 01 22:32:34.344103 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.344047    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm" event={"ID":"61e3e504-acf4-421d-b7c5-69b7bc9359bb","Type":"ContainerDied","Data":"83bad797dcef8c26d27229006cd7a8e27920ad7ed0142affd6664953d67b3484"}
Jul 01 22:32:34.344103 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.344076    2573 scope.go:117] "RemoveContainer" containerID="9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186"
Jul 01 22:32:34.345688 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.345663    2573 generic.go:356] "Generic (PLEG): container finished" podID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerID="04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313" exitCode=0
Jul 01 22:32:34.345688 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.345686    2573 generic.go:356] "Generic (PLEG): container finished" podID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerID="a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3" exitCode=0
Jul 01 22:32:34.345896 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.345709    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"
Jul 01 22:32:34.345896 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.345745    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr" event={"ID":"b047642c-5aaf-4278-8f57-f7b56b602c3f","Type":"ContainerDied","Data":"04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313"}
Jul 01 22:32:34.345896 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.345796    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr" event={"ID":"b047642c-5aaf-4278-8f57-f7b56b602c3f","Type":"ContainerDied","Data":"a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3"}
Jul 01 22:32:34.345896 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.345820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr" event={"ID":"b047642c-5aaf-4278-8f57-f7b56b602c3f","Type":"ContainerDied","Data":"038699b9d998810fd614eb57d52faf5f1914c265b1d8250864a2be3441b73491"}
Jul 01 22:32:34.353991 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.353960    2573 scope.go:117] "RemoveContainer" containerID="9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186"
Jul 01 22:32:34.354338 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:32:34.354228    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186\": container with ID starting with 9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186 not found: ID does not exist" containerID="9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186"
Jul 01 22:32:34.354338 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.354262    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186"} err="failed to get container status \"9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186\": rpc error: code = NotFound desc = could not find container \"9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186\": container with ID starting with 9f1a88df3699bf09acde413fd8b0f0462527e2349f12c9cb4620271089919186 not found: ID does not exist"
Jul 01 22:32:34.354338 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.354285    2573 scope.go:117] "RemoveContainer" containerID="04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313"
Jul 01 22:32:34.361977 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.361962    2573 scope.go:117] "RemoveContainer" containerID="a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3"
Jul 01 22:32:34.369393 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.369370    2573 scope.go:117] "RemoveContainer" containerID="04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313"
Jul 01 22:32:34.369650 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:32:34.369621    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313\": container with ID starting with 04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313 not found: ID does not exist" containerID="04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313"
Jul 01 22:32:34.369738 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.369648    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313"} err="failed to get container status \"04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313\": rpc error: code = NotFound desc = could not find container \"04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313\": container with ID starting with 04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313 not found: ID does not exist"
Jul 01 22:32:34.369738 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.369681    2573 scope.go:117] "RemoveContainer" containerID="a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3"
Jul 01 22:32:34.370046 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:32:34.370023    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3\": container with ID starting with a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3 not found: ID does not exist" containerID="a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3"
Jul 01 22:32:34.370177 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.370058    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3"} err="failed to get container status \"a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3\": rpc error: code = NotFound desc = could not find container \"a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3\": container with ID starting with a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3 not found: ID does not exist"
Jul 01 22:32:34.370177 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.370079    2573 scope.go:117] "RemoveContainer" containerID="04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313"
Jul 01 22:32:34.370386 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.370367    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313"} err="failed to get container status \"04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313\": rpc error: code = NotFound desc = could not find container \"04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313\": container with ID starting with 04844394cc467d279ed6696d60af58079e9044b4d19cd410488e6a31e6b76313 not found: ID does not exist"
Jul 01 22:32:34.370386 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.370384    2573 scope.go:117] "RemoveContainer" containerID="a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3"
Jul 01 22:32:34.370648 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.370628    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3"} err="failed to get container status \"a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3\": rpc error: code = NotFound desc = could not find container \"a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3\": container with ID starting with a294ce9563818107735d1163f5e70e78ec35c4e52b02cbaafac170641bcb7bc3 not found: ID does not exist"
Jul 01 22:32:34.372105 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.372084    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"]
Jul 01 22:32:34.376538 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.376519    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-kpnrr"]
Jul 01 22:32:34.387976 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.387958    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"]
Jul 01 22:32:34.394141 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:34.394122    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc56gcmm"]
Jul 01 22:32:35.590708 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:35.590676    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61e3e504-acf4-421d-b7c5-69b7bc9359bb" path="/var/lib/kubelet/pods/61e3e504-acf4-421d-b7c5-69b7bc9359bb/volumes"
Jul 01 22:32:35.591109 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:35.591067    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b047642c-5aaf-4278-8f57-f7b56b602c3f" path="/var/lib/kubelet/pods/b047642c-5aaf-4278-8f57-f7b56b602c3f/volumes"
Jul 01 22:32:39.688164 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688128    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xxfk/must-gather-6xkr6"]
Jul 01 22:32:39.688523 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688439    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerName="main"
Jul 01 22:32:39.688523 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688450    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerName="main"
Jul 01 22:32:39.688523 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688462    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerName="llm-d-routing-sidecar"
Jul 01 22:32:39.688523 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688468    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerName="llm-d-routing-sidecar"
Jul 01 22:32:39.688523 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688476    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e3e504-acf4-421d-b7c5-69b7bc9359bb" containerName="main"
Jul 01 22:32:39.688523 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688482    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e3e504-acf4-421d-b7c5-69b7bc9359bb" containerName="main"
Jul 01 22:32:39.688718 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688544    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e3e504-acf4-421d-b7c5-69b7bc9359bb" containerName="main"
Jul 01 22:32:39.688718 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688552    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerName="llm-d-routing-sidecar"
Jul 01 22:32:39.688718 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.688560    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b047642c-5aaf-4278-8f57-f7b56b602c3f" containerName="main"
Jul 01 22:32:39.693228 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.693209    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:32:39.697596 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.697567    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xxfk\"/\"kube-root-ca.crt\""
Jul 01 22:32:39.699066 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.699044    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8xxfk\"/\"default-dockercfg-f6prr\""
Jul 01 22:32:39.699299 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.699284    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xxfk\"/\"openshift-service-ca.crt\""
Jul 01 22:32:39.706748 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.706722    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xxfk/must-gather-6xkr6"]
Jul 01 22:32:39.808031 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.807998    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8264892-0b9b-43de-87aa-ff9869c25d81-must-gather-output\") pod \"must-gather-6xkr6\" (UID: \"e8264892-0b9b-43de-87aa-ff9869c25d81\") " pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:32:39.808148 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.808036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rggh2\" (UniqueName: \"kubernetes.io/projected/e8264892-0b9b-43de-87aa-ff9869c25d81-kube-api-access-rggh2\") pod \"must-gather-6xkr6\" (UID: \"e8264892-0b9b-43de-87aa-ff9869c25d81\") " pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:32:39.908644 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.908618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8264892-0b9b-43de-87aa-ff9869c25d81-must-gather-output\") pod \"must-gather-6xkr6\" (UID: \"e8264892-0b9b-43de-87aa-ff9869c25d81\") " pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:32:39.908736 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.908648    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rggh2\" (UniqueName: \"kubernetes.io/projected/e8264892-0b9b-43de-87aa-ff9869c25d81-kube-api-access-rggh2\") pod \"must-gather-6xkr6\" (UID: \"e8264892-0b9b-43de-87aa-ff9869c25d81\") " pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:32:39.908976 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.908950    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8264892-0b9b-43de-87aa-ff9869c25d81-must-gather-output\") pod \"must-gather-6xkr6\" (UID: \"e8264892-0b9b-43de-87aa-ff9869c25d81\") " pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:32:39.918050 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:39.918031    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rggh2\" (UniqueName: \"kubernetes.io/projected/e8264892-0b9b-43de-87aa-ff9869c25d81-kube-api-access-rggh2\") pod \"must-gather-6xkr6\" (UID: \"e8264892-0b9b-43de-87aa-ff9869c25d81\") " pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:32:40.004431 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:40.004375    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:32:40.131259 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:40.131232    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xxfk/must-gather-6xkr6"]
Jul 01 22:32:40.133274 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:32:40.133242    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8264892_0b9b_43de_87aa_ff9869c25d81.slice/crio-aa643e7d58f44a6162a387870d5ad817fc9cb67a5c12b2e7511de1f1aae56ddf WatchSource:0}: Error finding container aa643e7d58f44a6162a387870d5ad817fc9cb67a5c12b2e7511de1f1aae56ddf: Status 404 returned error can't find the container with id aa643e7d58f44a6162a387870d5ad817fc9cb67a5c12b2e7511de1f1aae56ddf
Jul 01 22:32:40.135078 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:40.135056    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 22:32:40.375321 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:40.375290    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xxfk/must-gather-6xkr6" event={"ID":"e8264892-0b9b-43de-87aa-ff9869c25d81","Type":"ContainerStarted","Data":"aa643e7d58f44a6162a387870d5ad817fc9cb67a5c12b2e7511de1f1aae56ddf"}
Jul 01 22:32:47.416356 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:47.416319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xxfk/must-gather-6xkr6" event={"ID":"e8264892-0b9b-43de-87aa-ff9869c25d81","Type":"ContainerStarted","Data":"c1e6feb87a72cdf1576ce0673bdbae5e864901e80460751f11a24b2f5e103e37"}
Jul 01 22:32:47.416765 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:47.416371    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xxfk/must-gather-6xkr6" event={"ID":"e8264892-0b9b-43de-87aa-ff9869c25d81","Type":"ContainerStarted","Data":"d3a4e2405456a3439e290de190e2609fea2371433f2a309ee36c26f182da4eb0"}
Jul 01 22:32:47.437550 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:47.437499    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8xxfk/must-gather-6xkr6" podStartSLOduration=2.257634694 podStartE2EDuration="8.43748637s" totalImagesPullingTime="6.179851676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 22:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:32:47.437317307 +0000 UTC m=+6400.348651934" watchObservedRunningTime="2026-07-01 22:32:47.43748637 +0000 UTC m=+6400.348821083"
Jul 01 22:32:55.648452 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:55.648419    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:32:57.213415 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:57.213384    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:32:58.765020 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:32:58.764986    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:00.272207 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:00.272177    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:01.781498 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:01.781471    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:03.358873 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:03.358822    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:04.837176 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:04.837149    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:06.315779 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:06.315748    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:07.938755 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:07.938719    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:09.520584 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:09.520555    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:11.010420 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:11.010370    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:12.597626 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:12.597598    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:14.096743 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:14.096711    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:15.594446 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:15.594417    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:17.074772 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:17.074738    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:18.545414 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:18.545381    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k_b1874861-4e64-470f-8964-44c6e8fd5683/main/0.log"
Jul 01 22:33:20.748804 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:20.748772    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcd57767c-rwpfk_1561f8b3-58e2-428a-98e1-5e2d115dd1ff/router/0.log"
Jul 01 22:33:22.424709 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:22.424679    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcd57767c-rwpfk_1561f8b3-58e2-428a-98e1-5e2d115dd1ff/router/0.log"
Jul 01 22:33:24.085219 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:24.085189    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcd57767c-rwpfk_1561f8b3-58e2-428a-98e1-5e2d115dd1ff/router/0.log"
Jul 01 22:33:25.766042 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:25.766007    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcd57767c-rwpfk_1561f8b3-58e2-428a-98e1-5e2d115dd1ff/router/0.log"
Jul 01 22:33:26.988473 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:26.988444    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:28.278653 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:28.278620    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:29.629992 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:29.629959    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:30.893876 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:30.893830    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:32.213423 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:32.213395    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:33.478967 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:33.478935    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:34.780204 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:34.780177    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:36.071376 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:36.071339    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:37.411840 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:37.411787    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:38.685541 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:38.685513    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:39.970056 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:39.970022    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:41.686696 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:41.686664    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8264892-0b9b-43de-87aa-ff9869c25d81" containerID="d3a4e2405456a3439e290de190e2609fea2371433f2a309ee36c26f182da4eb0" exitCode=0
Jul 01 22:33:41.687122 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:41.686733    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xxfk/must-gather-6xkr6" event={"ID":"e8264892-0b9b-43de-87aa-ff9869c25d81","Type":"ContainerDied","Data":"d3a4e2405456a3439e290de190e2609fea2371433f2a309ee36c26f182da4eb0"}
Jul 01 22:33:41.687122 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:41.687031    2573 scope.go:117] "RemoveContainer" containerID="d3a4e2405456a3439e290de190e2609fea2371433f2a309ee36c26f182da4eb0"
Jul 01 22:33:41.764214 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:41.764193    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8xxfk_must-gather-6xkr6_e8264892-0b9b-43de-87aa-ff9869c25d81/gather/0.log"
Jul 01 22:33:42.629740 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.629699    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z7v9q/must-gather-h9gxr"]
Jul 01 22:33:42.634328 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.634304    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z7v9q/must-gather-h9gxr"
Jul 01 22:33:42.637178 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.637159    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z7v9q\"/\"openshift-service-ca.crt\""
Jul 01 22:33:42.637259 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.637198    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z7v9q\"/\"default-dockercfg-pqvjh\""
Jul 01 22:33:42.637685 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.637667    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z7v9q\"/\"kube-root-ca.crt\""
Jul 01 22:33:42.645097 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.645077    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z7v9q/must-gather-h9gxr"]
Jul 01 22:33:42.755828 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.755804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8f169930-d9ae-4d32-a11f-a0b3c67dd309-must-gather-output\") pod \"must-gather-h9gxr\" (UID: \"8f169930-d9ae-4d32-a11f-a0b3c67dd309\") " pod="openshift-must-gather-z7v9q/must-gather-h9gxr"
Jul 01 22:33:42.756162 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.755836    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5wd\" (UniqueName: \"kubernetes.io/projected/8f169930-d9ae-4d32-a11f-a0b3c67dd309-kube-api-access-tr5wd\") pod \"must-gather-h9gxr\" (UID: \"8f169930-d9ae-4d32-a11f-a0b3c67dd309\") " pod="openshift-must-gather-z7v9q/must-gather-h9gxr"
Jul 01 22:33:42.856661 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.856629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8f169930-d9ae-4d32-a11f-a0b3c67dd309-must-gather-output\") pod \"must-gather-h9gxr\" (UID: \"8f169930-d9ae-4d32-a11f-a0b3c67dd309\") " pod="openshift-must-gather-z7v9q/must-gather-h9gxr"
Jul 01 22:33:42.856661 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.856664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5wd\" (UniqueName: \"kubernetes.io/projected/8f169930-d9ae-4d32-a11f-a0b3c67dd309-kube-api-access-tr5wd\") pod \"must-gather-h9gxr\" (UID: \"8f169930-d9ae-4d32-a11f-a0b3c67dd309\") " pod="openshift-must-gather-z7v9q/must-gather-h9gxr"
Jul 01 22:33:42.856997 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.856977    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8f169930-d9ae-4d32-a11f-a0b3c67dd309-must-gather-output\") pod \"must-gather-h9gxr\" (UID: \"8f169930-d9ae-4d32-a11f-a0b3c67dd309\") " pod="openshift-must-gather-z7v9q/must-gather-h9gxr"
Jul 01 22:33:42.866605 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.866584    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5wd\" (UniqueName: \"kubernetes.io/projected/8f169930-d9ae-4d32-a11f-a0b3c67dd309-kube-api-access-tr5wd\") pod \"must-gather-h9gxr\" (UID: \"8f169930-d9ae-4d32-a11f-a0b3c67dd309\") " pod="openshift-must-gather-z7v9q/must-gather-h9gxr"
Jul 01 22:33:42.944006 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:42.943952    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z7v9q/must-gather-h9gxr"
Jul 01 22:33:43.073324 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:43.073294    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z7v9q/must-gather-h9gxr"]
Jul 01 22:33:43.077017 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:33:43.076004    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f169930_d9ae_4d32_a11f_a0b3c67dd309.slice/crio-93d202c3d74da0623aa04f0be7d1ec5737a8970aaa70a0c21581ff28c079f466 WatchSource:0}: Error finding container 93d202c3d74da0623aa04f0be7d1ec5737a8970aaa70a0c21581ff28c079f466: Status 404 returned error can't find the container with id 93d202c3d74da0623aa04f0be7d1ec5737a8970aaa70a0c21581ff28c079f466
Jul 01 22:33:43.698006 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:43.697966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7v9q/must-gather-h9gxr" event={"ID":"8f169930-d9ae-4d32-a11f-a0b3c67dd309","Type":"ContainerStarted","Data":"93d202c3d74da0623aa04f0be7d1ec5737a8970aaa70a0c21581ff28c079f466"}
Jul 01 22:33:44.705072 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:44.705028    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7v9q/must-gather-h9gxr" event={"ID":"8f169930-d9ae-4d32-a11f-a0b3c67dd309","Type":"ContainerStarted","Data":"2a5d56f43127e48fb0f3269dac7b8117fb647c39b09ea835d05af7c898c81b84"}
Jul 01 22:33:44.705564 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:44.705092    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7v9q/must-gather-h9gxr" event={"ID":"8f169930-d9ae-4d32-a11f-a0b3c67dd309","Type":"ContainerStarted","Data":"0655c42b36ac629608f79f6eb3afef1defe9800ac822d43673ec5c2bb2d879b0"}
Jul 01 22:33:44.727174 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:44.727126    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z7v9q/must-gather-h9gxr" podStartSLOduration=2.009545593 podStartE2EDuration="2.727105631s" totalImagesPullingTime="717.560038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 22:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:33:44.725700207 +0000 UTC m=+6457.637034830" watchObservedRunningTime="2026-07-01 22:33:44.727105631 +0000 UTC m=+6457.638440259"
Jul 01 22:33:45.399943 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:45.399914    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zwrwf_6eb902c9-58f4-4714-b5c4-3259d67349f9/global-pull-secret-syncer/0.log"
Jul 01 22:33:45.512827 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:45.512798    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n8ht4_beef61b9-6bfd-4bb2-ade2-f535f587aa12/konnectivity-agent/0.log"
Jul 01 22:33:45.619873 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:45.619824    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-219.ec2.internal_75bb94aa8471c122d1bbd3ff2e95e5c0/haproxy/0.log"
Jul 01 22:33:47.465715 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.465679    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-8xxfk/must-gather-6xkr6"]
Jul 01 22:33:47.468411 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.468386    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-8xxfk/must-gather-6xkr6"]
Jul 01 22:33:47.469010 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.468611    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-8xxfk/must-gather-6xkr6" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" containerName="copy" containerID="cri-o://c1e6feb87a72cdf1576ce0673bdbae5e864901e80460751f11a24b2f5e103e37" gracePeriod=2
Jul 01 22:33:47.473801 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:33:47.473776    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6xkr6\" is forbidden: User \"system:node:ip-10-0-138-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8xxfk\": no relationship found between node 'ip-10-0-138-219.ec2.internal' and this object" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:33:47.595526 ip-10-0-138-219 kubenswrapper[2573]: E0701 22:33:47.595475    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6xkr6\" is forbidden: User \"system:node:ip-10-0-138-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8xxfk\": no relationship found between node 'ip-10-0-138-219.ec2.internal' and this object" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:33:47.730238 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.730147    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8xxfk_must-gather-6xkr6_e8264892-0b9b-43de-87aa-ff9869c25d81/copy/0.log"
Jul 01 22:33:47.730954 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.730919    2573 generic.go:356] "Generic (PLEG): container finished" podID="e8264892-0b9b-43de-87aa-ff9869c25d81" containerID="c1e6feb87a72cdf1576ce0673bdbae5e864901e80460751f11a24b2f5e103e37" exitCode=143
Jul 01 22:33:47.844917 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.839599    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8xxfk_must-gather-6xkr6_e8264892-0b9b-43de-87aa-ff9869c25d81/copy/0.log"
Jul 01 22:33:47.844917 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.842613    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:33:47.906804 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.906187    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8264892-0b9b-43de-87aa-ff9869c25d81-must-gather-output\") pod \"e8264892-0b9b-43de-87aa-ff9869c25d81\" (UID: \"e8264892-0b9b-43de-87aa-ff9869c25d81\") "
Jul 01 22:33:47.906804 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.906250    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rggh2\" (UniqueName: \"kubernetes.io/projected/e8264892-0b9b-43de-87aa-ff9869c25d81-kube-api-access-rggh2\") pod \"e8264892-0b9b-43de-87aa-ff9869c25d81\" (UID: \"e8264892-0b9b-43de-87aa-ff9869c25d81\") "
Jul 01 22:33:47.918701 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.918666    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8264892-0b9b-43de-87aa-ff9869c25d81-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e8264892-0b9b-43de-87aa-ff9869c25d81" (UID: "e8264892-0b9b-43de-87aa-ff9869c25d81"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 22:33:47.925268 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:47.925216    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8264892-0b9b-43de-87aa-ff9869c25d81-kube-api-access-rggh2" (OuterVolumeSpecName: "kube-api-access-rggh2") pod "e8264892-0b9b-43de-87aa-ff9869c25d81" (UID: "e8264892-0b9b-43de-87aa-ff9869c25d81"). InnerVolumeSpecName "kube-api-access-rggh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 22:33:48.007706 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:48.007596    2573 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8264892-0b9b-43de-87aa-ff9869c25d81-must-gather-output\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:33:48.007706 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:48.007634    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rggh2\" (UniqueName: \"kubernetes.io/projected/e8264892-0b9b-43de-87aa-ff9869c25d81-kube-api-access-rggh2\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\""
Jul 01 22:33:48.739921 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:48.739887    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8xxfk_must-gather-6xkr6_e8264892-0b9b-43de-87aa-ff9869c25d81/copy/0.log"
Jul 01 22:33:48.740475 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:48.740454    2573 scope.go:117] "RemoveContainer" containerID="c1e6feb87a72cdf1576ce0673bdbae5e864901e80460751f11a24b2f5e103e37"
Jul 01 22:33:48.740657 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:48.740638    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xxfk/must-gather-6xkr6"
Jul 01 22:33:48.761337 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:48.759627    2573 scope.go:117] "RemoveContainer" containerID="d3a4e2405456a3439e290de190e2609fea2371433f2a309ee36c26f182da4eb0"
Jul 01 22:33:49.594279 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:49.594237    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" path="/var/lib/kubelet/pods/e8264892-0b9b-43de-87aa-ff9869c25d81/volumes"
Jul 01 22:33:50.824340 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:50.824308    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-f448h_4de5123a-602a-438b-8683-4b39c6cf2c62/manager/0.log"
Jul 01 22:33:52.335229 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:52.335191    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-6xhzb_75c5a5b9-1fc9-4382-bf2e-ff152ad5a70d/cluster-monitoring-operator/0.log"
Jul 01 22:33:52.729618 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:52.729568    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fzt68_cbe17f7a-6da6-4b5a-bcc2-380ae08485b0/node-exporter/0.log"
Jul 01 22:33:52.756995 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:52.756942    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fzt68_cbe17f7a-6da6-4b5a-bcc2-380ae08485b0/kube-rbac-proxy/0.log"
Jul 01 22:33:52.784742 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:52.784715    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fzt68_cbe17f7a-6da6-4b5a-bcc2-380ae08485b0/init-textfile/0.log"
Jul 01 22:33:53.151432 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:53.151403    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-gh4cp_f0c0783c-df02-47db-8df1-0bf67b5d7b19/prometheus-operator-admission-webhook/0.log"
Jul 01 22:33:54.520722 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.520680    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-fhkqt_3ef16bee-7f4f-43cf-a467-ea42c7b6b2a0/networking-console-plugin/0.log"
Jul 01 22:33:54.835737 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.835691    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"]
Jul 01 22:33:54.836173 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.836150    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" containerName="copy"
Jul 01 22:33:54.836354 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.836339    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" containerName="copy"
Jul 01 22:33:54.836504 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.836490    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" containerName="gather"
Jul 01 22:33:54.836606 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.836595    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" containerName="gather"
Jul 01 22:33:54.836811 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.836796    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" containerName="copy"
Jul 01 22:33:54.836940 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.836928    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8264892-0b9b-43de-87aa-ff9869c25d81" containerName="gather"
Jul 01 22:33:54.841298 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.841277    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.853124 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.853100    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"]
Jul 01 22:33:54.873196 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.873168    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-sys\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.873332 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.873214    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-proc\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.873332 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.873245    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzmj\" (UniqueName: \"kubernetes.io/projected/4c7a67a3-8c14-4b78-87e3-2203daa1abec-kube-api-access-xxzmj\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.873464 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.873353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-podres\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.873464 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.873383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-lib-modules\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.974436 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.974401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-proc\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.974436 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.974444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzmj\" (UniqueName: \"kubernetes.io/projected/4c7a67a3-8c14-4b78-87e3-2203daa1abec-kube-api-access-xxzmj\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.974665 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.974514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-podres\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.974665 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.974533    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-proc\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.974665 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.974544    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-lib-modules\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.974665 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.974592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-sys\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.974665 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.974656    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-lib-modules\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.974665 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.974664    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-sys\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.974892 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.974658    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c7a67a3-8c14-4b78-87e3-2203daa1abec-podres\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:54.983477 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:54.983439    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzmj\" (UniqueName: \"kubernetes.io/projected/4c7a67a3-8c14-4b78-87e3-2203daa1abec-kube-api-access-xxzmj\") pod \"perf-node-gather-daemonset-dwd74\" (UID: \"4c7a67a3-8c14-4b78-87e3-2203daa1abec\") " pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:55.046029 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:55.045998    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/2.log"
Jul 01 22:33:55.054215 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:55.054188    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-r5d2x_4861b515-af80-46a0-a870-438254fa8c38/console-operator/3.log"
Jul 01 22:33:55.153083 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:55.152988    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:55.312633 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:55.312606    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"]
Jul 01 22:33:55.313032 ip-10-0-138-219 kubenswrapper[2573]: W0701 22:33:55.313009    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4c7a67a3_8c14_4b78_87e3_2203daa1abec.slice/crio-7e1cebb5f6e572f6a2b9f91c03ae504617b6da25acd2ccdb47a8dd4c89ac0926 WatchSource:0}: Error finding container 7e1cebb5f6e572f6a2b9f91c03ae504617b6da25acd2ccdb47a8dd4c89ac0926: Status 404 returned error can't find the container with id 7e1cebb5f6e572f6a2b9f91c03ae504617b6da25acd2ccdb47a8dd4c89ac0926
Jul 01 22:33:55.784689 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:55.784606    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74" event={"ID":"4c7a67a3-8c14-4b78-87e3-2203daa1abec","Type":"ContainerStarted","Data":"bf92138266c2d56157fd23738cea0d48f5375bcc945f700f1c0a14284af77d01"}
Jul 01 22:33:55.784689 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:55.784662    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74" event={"ID":"4c7a67a3-8c14-4b78-87e3-2203daa1abec","Type":"ContainerStarted","Data":"7e1cebb5f6e572f6a2b9f91c03ae504617b6da25acd2ccdb47a8dd4c89ac0926"}
Jul 01 22:33:55.785186 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:55.784747    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:33:55.803135 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:55.803089    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74" podStartSLOduration=1.8030759490000001 podStartE2EDuration="1.803075949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 22:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 22:33:55.802541123 +0000 UTC m=+6468.713875749" watchObservedRunningTime="2026-07-01 22:33:55.803075949 +0000 UTC m=+6468.714410575"
Jul 01 22:33:56.207822 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:56.207787    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-69d9f679db-gd5rv_d6b4a894-2b09-4c5b-8df9-50f5f5e64e4e/volume-data-source-validator/0.log"
Jul 01 22:33:57.220502 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:57.220478    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fj468_f8705dc8-bdeb-4766-b015-b5b864ca8ae7/dns/0.log"
Jul 01 22:33:57.244456 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:57.244415    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fj468_f8705dc8-bdeb-4766-b015-b5b864ca8ae7/kube-rbac-proxy/0.log"
Jul 01 22:33:57.340041 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:57.340010    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-llqwv_f4938d98-b1e1-4724-97e2-0e948765bd87/dns-node-resolver/0.log"
Jul 01 22:33:57.907401 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:57.907367    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ffv5j_634951d8-071a-4335-9c1d-ef274d90307e/node-ca/0.log"
Jul 01 22:33:59.630091 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:33:59.630006    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcd57767c-rwpfk_1561f8b3-58e2-428a-98e1-5e2d115dd1ff/router/0.log"
Jul 01 22:34:00.262100 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:00.262052    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s6d8g_90dd030d-72b9-4b93-ae9b-24aeec54eca6/serve-healthcheck-canary/0.log"
Jul 01 22:34:00.714190 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:00.714155    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-zs75b_97bad7a1-1048-413b-a19c-4768dfca09e5/insights-operator/0.log"
Jul 01 22:34:00.717776 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:00.717744    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-zs75b_97bad7a1-1048-413b-a19c-4768dfca09e5/insights-operator/1.log"
Jul 01 22:34:00.814968 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:00.814923    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sk28n_f4ad4488-3e80-4ce5-84c3-1b6524053abe/kube-rbac-proxy/0.log"
Jul 01 22:34:00.834618 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:00.834591    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sk28n_f4ad4488-3e80-4ce5-84c3-1b6524053abe/exporter/0.log"
Jul 01 22:34:00.858770 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:00.858745    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sk28n_f4ad4488-3e80-4ce5-84c3-1b6524053abe/extractor/0.log"
Jul 01 22:34:01.801949 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:01.801921    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z7v9q/perf-node-gather-daemonset-dwd74"
Jul 01 22:34:03.858275 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:03.858245    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-55f5fcc78b-k2lxv_f9889cdb-b6e8-433c-96fb-e67296d1279b/manager/0.log"
Jul 01 22:34:05.249535 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:05.249506    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zqr6x_e8442666-24ad-40ef-ac08-ced96e706f5a/s3-init/0.log"
Jul 01 22:34:05.267714 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:05.267693    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zqr6x_e8442666-24ad-40ef-ac08-ced96e706f5a/download-hf-model/0.log"
Jul 01 22:34:06.997396 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:06.997315    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcd57767c-rwpfk_1561f8b3-58e2-428a-98e1-5e2d115dd1ff/router/0.log"
Jul 01 22:34:08.799665 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:08.799632    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bcd57767c-rwpfk_1561f8b3-58e2-428a-98e1-5e2d115dd1ff/router/0.log"
Jul 01 22:34:10.610035 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:10.610003    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-2hghb_b484c66a-201d-48bd-9925-25bc2c2170af/migrator/0.log"
Jul 01 22:34:10.641160 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:10.641131    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-2hghb_b484c66a-201d-48bd-9925-25bc2c2170af/graceful-termination/0.log"
Jul 01 22:34:10.992783 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:10.992684    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-6j4dv_e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc/kube-storage-version-migrator-operator/1.log"
Jul 01 22:34:10.994389 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:10.994358    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-6j4dv_e8fc71dc-1f85-422e-b6a9-2cd2c4fe28dc/kube-storage-version-migrator-operator/0.log"
Jul 01 22:34:11.946741 ip-10-0-138-219 kubenswrapper[2573]: I0701 22:34:11.946647    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-85vx9_82ea0a51-b205-4ba9-8356-a96baf3f0de6/kube-multus/0.log"