Jul 09 18:02:25.655139 ip-10-0-129-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 18:02:25.655149 ip-10-0-129-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 18:02:25.655159 ip-10-0-129-87 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 18:02:25.655550 ip-10-0-129-87 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 18:02:35.827253 ip-10-0-129-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 18:02:35.827295 ip-10-0-129-87 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f989f5a579ca4a50938dcbd845b7ce47 --
Jul 09 18:04:39.046454 ip-10-0-129-87 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.549583    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558719    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558732    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558736    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558739    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558742    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558745    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558748    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:40.600397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558751    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558756    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558760    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558762    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558765    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558770    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558773    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558777    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558780    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558783    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558785    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558793    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558796    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558799    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558802    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558805    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558807    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558811    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558813    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:40.602256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558816    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558819    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558822    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558824    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558827    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558829    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558832    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558834    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558837    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558841    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558844    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558847    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558850    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558853    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558855    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558858    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558860    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558863    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558865    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558868    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:40.602872 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558870    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558873    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558875    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558878    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558881    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558883    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558886    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558888    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558891    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558894    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558896    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558899    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558918    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558921    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558923    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558926    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558929    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558932    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558934    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558937    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:40.603446 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558939    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558942    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558945    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558948    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558950    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558955    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558957    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558960    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558963    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558965    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558970    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558973    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558977    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558981    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558984    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558987    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558990    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558993    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558997    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:40.604023 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.558999    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559002    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559004    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559007    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559009    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559012    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559014    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559017    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559020    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559022    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559024    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559027    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559029    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559032    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559734    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559740    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559742    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559746    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559748    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:40.604572 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559753    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559757    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559762    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559765    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559769    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559772    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559775    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559778    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559782    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559784    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559787    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559790    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559793    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559795    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559798    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559801    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559804    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559806    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559809    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:40.605244 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559812    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559814    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559817    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559819    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559822    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559824    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559827    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559830    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559832    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559835    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559838    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559841    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559843    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559846    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559849    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559851    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559854    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559856    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559859    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559861    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:40.605783 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559863    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559866    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559868    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559871    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559873    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559875    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559878    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559881    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559883    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559885    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559888    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559891    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559893    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559895    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559898    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559925    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559928    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559930    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559933    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559936    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:40.606397 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559940    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559943    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559946    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559949    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559952    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559957    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559960    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559964    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559967    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559970    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559973    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559975    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559978    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559981    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559983    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559987    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559989    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559992    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559995    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.559997    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:40.606944 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560000    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560002    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560005    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560007    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560010    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560013    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560015    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560018    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560020    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560022    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560025    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560028    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560030    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560033    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560036    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560451    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560456    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560461    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560465    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:40.607520 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560474    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560477    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560480    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560482    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560485    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560487    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560490    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560492    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560495    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560497    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560500    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560503    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560505    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560508    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560510    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560513    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560515    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560517    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560520    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560523    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:40.608032 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560525    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560528    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560530    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560533    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560535    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560538    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560540    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560543    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560547    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560557    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560561    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560564    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560566    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560575    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560579    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560582    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560585    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560588    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560591    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:40.608622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560594    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560597    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560599    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560602    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560604    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560607    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560610    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560613    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560615    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560618    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560620    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560623    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560625    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560628    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560630    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560633    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560635    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560637    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560640    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560642    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:40.609242 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560645    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560647    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560650    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560654    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560657    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560659    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560662    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560670    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560675    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560679    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560683    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560686    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560690    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560692    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560696    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560698    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560701    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560704    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560707    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:40.609797 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560710    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560712    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560715    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560718    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560721    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560723    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560726    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560729    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560732    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560734    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560737    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560739    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560742    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560744    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560747    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560750    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.560754    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560835    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560842    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560849    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560853    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:04:40.610471 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560858    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560861    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560865    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560869    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560873    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560875    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560879    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560882    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560885    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560888    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560891    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560896    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560899    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560915    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560920    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560923    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560927    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560931    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560934    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560938    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560941    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560944    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560947    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560951    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:04:40.611084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560954    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560957    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560960    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560965    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560969    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560972    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560975    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560977    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560981    2565 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560984    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560989    2565 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560992    2565 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560995    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.560998    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561001    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561005    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561008    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561011    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561014    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561018    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561021    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561024    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561027    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561030    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561033    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:04:40.611998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561036    2565 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561039    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561043    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561046    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561049    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561052    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561055    2565 flags.go:64] FLAG: --help="false"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561057    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-129-87.ec2.internal"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561060    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561063    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561068    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561071    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561074    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561077    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561080    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561083    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561086    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561089    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561092    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561095    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561098    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561101    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561104    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561107    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:04:40.613193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561110    2565 flags.go:64] FLAG: --lock-file=""
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561112    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561117    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561120    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561125    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561128    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561131    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561133    2565 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561136    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561139    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561143    2565 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561145    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561150    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561153    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561156    2565 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561159    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561162    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561165    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561169    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561172    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561176    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561179    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561187    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561189    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561192    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:04:40.614448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561195    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561205    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561211    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561214    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561217    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561220    2565 flags.go:64] FLAG: --port="10250"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561223    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561225    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a213289c1a183d44"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561229    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561234    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561237    2565 flags.go:64] FLAG: --register-node="true"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561240    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561244    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561247    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561250    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561253    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561257    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561260    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561263    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561266    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561269    2565 flags.go:64] FLAG: --runonce="false"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561271    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561275    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561277    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561280    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:04:40.615701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561283    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561288    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561291    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561294    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561297    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561300    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561302    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561305    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561308    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561317    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561320    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561333    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561336    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561338    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561342    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561345    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561349    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561352    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561355    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561358    2565 flags.go:64] FLAG: --v="2"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561362    2565 flags.go:64] FLAG: --version="false"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561366    2565 flags.go:64] FLAG: --vmodule=""
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561369    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.561373    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561512    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:40.617129 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561516    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561521    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561524    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561527    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561530    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561533    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561536    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561538    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561543    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561545    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561548    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561551    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561553    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561556    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561559    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561561    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561564    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561567    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561571    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561575    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:40.752853 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561577    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:40.641776 ip-10-0-129-87 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561581    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561585    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561590    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561593    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561596    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561598    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561601    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561603    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561606    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561609    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561611    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561614    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561616    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561619    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561622    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561624    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561627    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561629    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:40.754053 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561632    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561635    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561638    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561641    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561643    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561646    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561648    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561651    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561653    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561656    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561658    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561661    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561664    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561666    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561669    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561671    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561676    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561678    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561681    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561683    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:40.754732 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561685    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561688    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561691    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561693    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561696    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561698    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561701    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561703    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561706    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561709    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561711    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561714    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561716    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561720    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561723    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561734    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561736    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561739    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561742    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561744    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:40.755519 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561747    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561749    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561752    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561755    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561761    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561764    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561767    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561770    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561774    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561776    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561779    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561781    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561784    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561787    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561790    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561792    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561795    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561797    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:40.756269 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.561800    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.562275    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.569992    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.570023    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.570045    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.570051    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570127    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570132    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570136    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570140    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570145    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570148    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570151    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570154    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:40.756854 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570157    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570160    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570163    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570165    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570168    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570171    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570173    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570176    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570179    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570181    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570184    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570187    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570189    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570193    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570196    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570198    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570201    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570204    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570207    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570210    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:40.757396 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570212    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570217    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570221    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570224    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570227    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570230    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570233    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570235    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570238    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570240    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570243    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570245    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570248    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570250    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570253    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570256    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570258    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570261    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570263    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:40.758041 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570266    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570268    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570271    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570273    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570276    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570278    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570286    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570289    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570291    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570294    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570297    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570300    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570302    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570305    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570307    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570311    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570313    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570317    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570319    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570322    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:40.760102 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570324    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570327    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570329    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570332    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570335    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570337    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570340    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570342    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570346    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570349    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570352    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570355    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570358    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570361    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570363    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570366    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570368    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570371    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570375    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570377    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:40.760992 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570380    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570383    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570385    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570388    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570391    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570393    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570396    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570398    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570402    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570404    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570407    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570409    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.570414    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570538    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570544    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:40.785456 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570547    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570550    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570553    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570556    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570558    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570561    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570563    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570566    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570569    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570572    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570574    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570578    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570583    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570585    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570588    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570593    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570598    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570600    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570603    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570605    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:40.789962 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570608    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570611    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570613    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570616    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570618    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570621    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570624    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570626    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570629    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570633    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570636    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570639    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570642    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570645    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570648    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570651    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570653    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570656    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570659    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:40.790655 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570662    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570665    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570668    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570670    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570673    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570675    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570678    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570680    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570686    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570691    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570693    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570696    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570699    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570701    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570704    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570706    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570709    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570712    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570714    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570717    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:40.791291 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570720    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570722    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570725    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570727    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570730    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570732    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570735    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570737    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570740    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570742    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570745    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570747    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570750    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570752    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570755    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570757    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570761    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570764    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570767    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:40.792015 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570770    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570773    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570777    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570780    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570782    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570785    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570788    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570790    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570793    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570795    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570798    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570800    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570803    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570806    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570809    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570811    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570814    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570816    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:40.792586 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:40.570819    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.570824    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.570971    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.572980    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.574265    2565 server.go:1019] "Starting client certificate rotation"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.574371    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.575183    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.578044    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.580725    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.597034    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.604321    2565 log.go:25] "Validated CRI v1 image API"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.605201    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:04:40.793122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.611992    2565 fs.go:135] Filesystem UUIDs: map[1a6ebeda-4b34-470f-af11-dbb3acc0e24a:/dev/nvme0n1p4 4b478bee-aacd-4ef3-be98-49dd4ca915e2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 09 18:04:40.793535 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.612020    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:04:40.793682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.618330    2565 manager.go:216] Machine: {Timestamp:2026-07-09 18:04:40.617451688 +0000 UTC m=+0.335714445 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100178 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cd5ee361c9fdc9e20b01cc024b700 SystemUUID:ec2cd5ee-361c-9fdc-9e20-b01cc024b700 BootID:f989f5a5-79ca-4a50-938d-cbd845b7ce47 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ba:d8:4d:44:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ba:d8:4d:44:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:c8:b3:9f:1a:6a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:04:40.793682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.618432    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:04:40.793682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.618514    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:04:40.793682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.621120    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:04:40.793682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.621156    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-87.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:04:40.793682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.621822    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:04:40.793682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.621833    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:04:40.793682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.621841    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.622477    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.623770    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.626822    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.626992    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.627029    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.627045    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.627056    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.627065    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.627073    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.627418    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.627434    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.627825    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.634444    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636203    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636231    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636870    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636885    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636891    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636897    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636918    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636924    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636932    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636938    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636945    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636951    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636959    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.636964    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.637673    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.637679    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.640814    2565 server.go:1268] "Started kubelet"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.640875    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.640918    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.641280    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.641859    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-87.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.645135    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.645196    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.645365    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.652469    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:04:40.794044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.653853    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.655325    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.655339    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.655401    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.655933    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.655989    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.655934    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.656163    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.656171    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.656761    2565 factory.go:55] Registering systemd factory
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.656773    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.656981    2565 factory.go:153] Registering CRI-O factory
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.656990    2565 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.657038    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.657064    2565 factory.go:103] Registering Raw factory
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.657079    2565 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.657565    2565 manager.go:349] Starting recovery of all containers
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.659948    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.660136    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.669806    2565 manager.go:354] Recovery completed
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.671098    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-87.ec2.internal.18c0b131f9c6c5b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-87.ec2.internal,UID:ip-10-0-129-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-87.ec2.internal,},FirstTimestamp:2026-07-09 18:04:40.640783795 +0000 UTC m=+0.359046552,LastTimestamp:2026-07-09 18:04:40.640783795 +0000 UTC m=+0.359046552,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-87.ec2.internal,}"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.674705    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.676711    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.676737    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.676748    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.677355    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.677364    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.677381    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.677357    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.679173    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.679199    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:40.795184 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.679211    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.679863    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-87.ec2.internal.18c0b131fbeb2e98  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-87.ec2.internal,UID:ip-10-0-129-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-87.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-87.ec2.internal,},FirstTimestamp:2026-07-09 18:04:40.676724376 +0000 UTC m=+0.394987130,LastTimestamp:2026-07-09 18:04:40.676724376 +0000 UTC m=+0.394987130,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-87.ec2.internal,}"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.684069    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9rcmj"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.684602    2565 policy_none.go:49] "None policy: Start"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.684615    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.684626    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.685871    2565 policy_none.go:47] "Start"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.689811    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9rcmj"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.690762    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-87.ec2.internal.18c0b131fbeb74a8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-87.ec2.internal,UID:ip-10-0-129-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-87.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-87.ec2.internal,},FirstTimestamp:2026-07-09 18:04:40.676742312 +0000 UTC m=+0.395005067,LastTimestamp:2026-07-09 18:04:40.676742312 +0000 UTC m=+0.395005067,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-87.ec2.internal,}"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.737549    2565 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.737597    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.737610    2565 server.go:87] "Starting device plugin registration server"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.737944    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.737957    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.738037    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.738167    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.738179    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.739744    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:04:40.796139 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.739799    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-87.ec2.internal\" not found"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.838595    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.839659    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.839692    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.839702    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.839728    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-87.ec2.internal"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.858170    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-87.ec2.internal"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.858190    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-87.ec2.internal\": node \"ip-10-0-129-87.ec2.internal\" not found"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.860594    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.861784    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.861815    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.861839    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.861872    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.866489    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:40.927742    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.962122    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"]
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.962189    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.964248    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.964277    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:40.965830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.964289    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:40.966522 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.966329    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:40.967326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.966882    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:40.967326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.966925    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:40.967326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.967127    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:40.967326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.967152    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:40.967326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.967165    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:40.967750 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.967586    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:40.967750 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.967611    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:40.967750 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.967620    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:40.968742 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.968617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:40.968742 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.968640    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:41.083562 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.969295    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:41.083562 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.969321    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:41.083562 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:40.969332    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:41.083562 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.004698    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-87.ec2.internal\" not found" node="ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.083562 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.009177    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-87.ec2.internal\" not found" node="ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.083562 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.028226    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Jul 09 18:04:41.083562 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.058554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7423fcc2f22fef5c1cd5088939f63a04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"7423fcc2f22fef5c1cd5088939f63a04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.083562 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.058582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7423fcc2f22fef5c1cd5088939f63a04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"7423fcc2f22fef5c1cd5088939f63a04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.083562 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.058600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1ab68799b6023da576fe8586ec62200-config\") pod \"kube-apiserver-proxy-ip-10-0-129-87.ec2.internal\" (UID: \"d1ab68799b6023da576fe8586ec62200\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.164888 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.129176    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Jul 09 18:04:41.164888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.159530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7423fcc2f22fef5c1cd5088939f63a04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"7423fcc2f22fef5c1cd5088939f63a04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.164888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.159567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1ab68799b6023da576fe8586ec62200-config\") pod \"kube-apiserver-proxy-ip-10-0-129-87.ec2.internal\" (UID: \"d1ab68799b6023da576fe8586ec62200\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.164888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.159592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7423fcc2f22fef5c1cd5088939f63a04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"7423fcc2f22fef5c1cd5088939f63a04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.164888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.159601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7423fcc2f22fef5c1cd5088939f63a04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"7423fcc2f22fef5c1cd5088939f63a04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.164888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.159643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1ab68799b6023da576fe8586ec62200-config\") pod \"kube-apiserver-proxy-ip-10-0-129-87.ec2.internal\" (UID: \"d1ab68799b6023da576fe8586ec62200\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.164888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.159661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7423fcc2f22fef5c1cd5088939f63a04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal\" (UID: \"7423fcc2f22fef5c1cd5088939f63a04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.262848 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.229500    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Jul 09 18:04:41.420457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.307002    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.420457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.312859    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.420457 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.329747    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Jul 09 18:04:41.543418 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.429984    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-87.ec2.internal\" not found"
Jul 09 18:04:41.543418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.526133    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:41.556532 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.556500    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.573470 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.573440    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:04:41.573635 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.573611    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:41.573674 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.573651    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:41.573708 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.573672    2565 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a764e0fb0d810454588d653421b3d422-b21387e8c19fe429.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.129.87:55242->54.156.100.98:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.573708 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.573702    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"
Jul 09 18:04:41.609648 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.609624    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:41.627795 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.627769    2565 apiserver.go:52] "Watching apiserver"
Jul 09 18:04:41.637232 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.637209    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:04:41.639992 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.639971    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6","openshift-cluster-node-tuning-operator/tuned-p2sr6","openshift-image-registry/node-ca-2ptrg","openshift-multus/multus-4kfcs","openshift-multus/multus-additional-cni-plugins-9lq7r","openshift-ovn-kubernetes/ovnkube-node-dlqjl","kube-system/global-pull-secret-syncer-lznbs","kube-system/konnectivity-agent-nh5dl","openshift-dns/node-resolver-4s6pg","openshift-multus/network-metrics-daemon-jn9vx","openshift-network-diagnostics/network-check-target-h2cxx","openshift-network-operator/iptables-alerter-lzh6n","kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal"]
Jul 09 18:04:41.641608 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.641591    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:41.641691 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.641670    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:41.642888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.642873    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.644327 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.644304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.647286 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.647266    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.651751 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.651731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.652292 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.652275    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:04:41.652400 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.652382    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:04:41.652552 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.652534    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.652659 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.652557    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.652659 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.652568    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.652659 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.652602    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:04:41.652659 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.652656    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.653059 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.653046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:41.653120 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.653104    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:04:41.654144 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.654130    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.655482 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.655469    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:41.655594 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.655580    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:41.655672 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.655652    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:41.656732 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.656717    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:04:41.658162 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.658148    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.659503 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.659478    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.660859 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.660841    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.662797 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.662775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-sysconfig\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.662898 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.662809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-host\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.662898 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.662839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-kubelet\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.662898 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.662865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-systemd-units\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.662898 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.662887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/250e9a05-21d4-45d5-9113-f4594095ffdb-env-overrides\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.663131 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.662923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dmd\" (UniqueName: \"kubernetes.io/projected/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-kube-api-access-m9dmd\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:41.663131 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.662947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d764999-f70b-4528-a03e-6e8eae6d6bae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.663131 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.662971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fe991f9-27b4-4d56-a779-014879464f58-dbus\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:41.663131 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:41.663131 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-cnibin\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.663131 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-var-lib-cni-multus\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.663355 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-run-systemd\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.663355 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-os-release\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.663355 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-modprobe-d\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.663484 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-etc-openvswitch\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.663484 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.663484 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-system-cni-dir\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.663629 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-var-lib-cni-bin\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.663849 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-systemd\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.663894 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-run\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.663894 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-tmp\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.663894 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-slash\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.663894 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:41.664118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-run-ovn\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.664118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d764999-f70b-4528-a03e-6e8eae6d6bae-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.664118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.664118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.663999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-sys\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.664118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/250e9a05-21d4-45d5-9113-f4594095ffdb-ovn-node-metrics-cert\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.664118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4991788b-de70-4f49-87f4-cad8cabd7fe3-konnectivity-ca\") pod \"konnectivity-agent-nh5dl\" (UID: \"4991788b-de70-4f49-87f4-cad8cabd7fe3\") " pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:04:41.664118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/364ccd55-cb96-46e5-a706-bcdf005bf715-serviceca\") pod \"node-ca-2ptrg\" (UID: \"364ccd55-cb96-46e5-a706-bcdf005bf715\") " pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.664118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vcx\" (UniqueName: \"kubernetes.io/projected/364ccd55-cb96-46e5-a706-bcdf005bf715-kube-api-access-d7vcx\") pod \"node-ca-2ptrg\" (UID: \"364ccd55-cb96-46e5-a706-bcdf005bf715\") " pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.664118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664094    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/506b3c80-5b84-408c-b9ed-55b660477968-multus-daemon-config\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664120    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-lib-modules\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-tuned\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-run-openvswitch\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-os-release\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-hostroot\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p6fb\" (UniqueName: \"kubernetes.io/projected/506b3c80-5b84-408c-b9ed-55b660477968-kube-api-access-2p6fb\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-node-log\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-log-socket\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhpg\" (UniqueName: \"kubernetes.io/projected/250e9a05-21d4-45d5-9113-f4594095ffdb-kube-api-access-xmhpg\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-multus-cni-dir\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-kubernetes\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-sysctl-conf\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-var-lib-kubelet\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.664498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664399    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/364ccd55-cb96-46e5-a706-bcdf005bf715-host\") pod \"node-ca-2ptrg\" (UID: \"364ccd55-cb96-46e5-a706-bcdf005bf715\") " pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-multus-socket-dir-parent\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664456    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-sysctl-d\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7b15ac-cf17-4a04-be09-eaadda984155-tmp-dir\") pod \"node-resolver-4s6pg\" (UID: \"3c7b15ac-cf17-4a04-be09-eaadda984155\") " pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlwtc\" (UniqueName: \"kubernetes.io/projected/3c7b15ac-cf17-4a04-be09-eaadda984155-kube-api-access-rlwtc\") pod \"node-resolver-4s6pg\" (UID: \"3c7b15ac-cf17-4a04-be09-eaadda984155\") " pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/506b3c80-5b84-408c-b9ed-55b660477968-cni-binary-copy\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-run-netns\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjgh\" (UniqueName: \"kubernetes.io/projected/7d764999-f70b-4528-a03e-6e8eae6d6bae-kube-api-access-9wjgh\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664633    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-var-lib-openvswitch\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fe991f9-27b4-4d56-a779-014879464f58-kubelet-config\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-system-cni-dir\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttzd\" (UniqueName: \"kubernetes.io/projected/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-kube-api-access-zttzd\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-cni-bin\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.665048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/250e9a05-21d4-45d5-9113-f4594095ffdb-ovnkube-script-lib\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3c7b15ac-cf17-4a04-be09-eaadda984155-hosts-file\") pod \"node-resolver-4s6pg\" (UID: \"3c7b15ac-cf17-4a04-be09-eaadda984155\") " pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-run-k8s-cni-cncf-io\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-var-lib-kubelet\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664802    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/250e9a05-21d4-45d5-9113-f4594095ffdb-ovnkube-config\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-multus-conf-dir\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-etc-kubernetes\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-cnibin\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d764999-f70b-4528-a03e-6e8eae6d6bae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-run-netns\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-cni-netd\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4991788b-de70-4f49-87f4-cad8cabd7fe3-agent-certs\") pod \"konnectivity-agent-nh5dl\" (UID: \"4991788b-de70-4f49-87f4-cad8cabd7fe3\") " pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:04:41.665632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.664972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-run-multus-certs\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.666755 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.666739    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pjxsz\""
Jul 09 18:04:41.667269 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.667255    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zhvng\""
Jul 09 18:04:41.667972 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.667959    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:04:41.668053 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.668006    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.668209 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.668195    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k4bhf\""
Jul 09 18:04:41.668373 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.668361    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:04:41.669050 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.669035    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8hzgk\""
Jul 09 18:04:41.669219 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.669201    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:04:41.679883 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.679860    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:04:41.680348 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.680311    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.680348 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.680342    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.680609 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.680559    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:04:41.680764 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.680749    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.680836 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.680765    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.681733 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.681686    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.690710 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.690697    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qwp2m\""
Jul 09 18:04:41.691800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.691781    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:04:41.691916 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.691844    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q27q5\""
Jul 09 18:04:41.691916 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.691858    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:04:41.692117 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.692102    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.692117 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.692112    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:04:41.692277 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.692141    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.692277 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.692230    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2tx24\""
Jul 09 18:04:41.692374 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.692279    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:04:41.692374 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.692295    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:04:41.692374 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.692344    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.693468 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.693442    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:59:40 +0000 UTC" deadline="2028-02-18 18:21:54.923469276 +0000 UTC"
Jul 09 18:04:41.693555 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.693469    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14136h17m13.230003648s"
Jul 09 18:04:41.693555 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.693487    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cq5v2\""
Jul 09 18:04:41.693681 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.693622    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j5g4d\""
Jul 09 18:04:41.704969 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.704952    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.718156 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.718135    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:41.739687 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.739491    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:41.756621 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.756600    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:04:41.765202 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-sysctl-d\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.765345 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7b15ac-cf17-4a04-be09-eaadda984155-tmp-dir\") pod \"node-resolver-4s6pg\" (UID: \"3c7b15ac-cf17-4a04-be09-eaadda984155\") " pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.765345 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlwtc\" (UniqueName: \"kubernetes.io/projected/3c7b15ac-cf17-4a04-be09-eaadda984155-kube-api-access-rlwtc\") pod \"node-resolver-4s6pg\" (UID: \"3c7b15ac-cf17-4a04-be09-eaadda984155\") " pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.765345 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/506b3c80-5b84-408c-b9ed-55b660477968-cni-binary-copy\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.765473 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-sysctl-d\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.765541 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-run-netns\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.765703 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-run-netns\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.765753 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7b15ac-cf17-4a04-be09-eaadda984155-tmp-dir\") pod \"node-resolver-4s6pg\" (UID: \"3c7b15ac-cf17-4a04-be09-eaadda984155\") " pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.765789 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjgh\" (UniqueName: \"kubernetes.io/projected/7d764999-f70b-4528-a03e-6e8eae6d6bae-kube-api-access-9wjgh\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.765789 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-var-lib-openvswitch\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.765865 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765806    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fe991f9-27b4-4d56-a779-014879464f58-kubelet-config\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:41.765865 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-device-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.765865 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-var-lib-openvswitch\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.766044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765861    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-system-cni-dir\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.766044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttzd\" (UniqueName: \"kubernetes.io/projected/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-kube-api-access-zttzd\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.766044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fe991f9-27b4-4d56-a779-014879464f58-kubelet-config\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:41.766044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-cni-bin\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.766044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/506b3c80-5b84-408c-b9ed-55b660477968-cni-binary-copy\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.766044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-system-cni-dir\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.766044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.765995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/250e9a05-21d4-45d5-9113-f4594095ffdb-ovnkube-script-lib\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.766044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-socket-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.766044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-cni-bin\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3c7b15ac-cf17-4a04-be09-eaadda984155-hosts-file\") pod \"node-resolver-4s6pg\" (UID: \"3c7b15ac-cf17-4a04-be09-eaadda984155\") " pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-run-k8s-cni-cncf-io\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-var-lib-kubelet\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3c7b15ac-cf17-4a04-be09-eaadda984155-hosts-file\") pod \"node-resolver-4s6pg\" (UID: \"3c7b15ac-cf17-4a04-be09-eaadda984155\") " pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/250e9a05-21d4-45d5-9113-f4594095ffdb-ovnkube-config\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-multus-conf-dir\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-etc-kubernetes\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-var-lib-kubelet\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-run-k8s-cni-cncf-io\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-cnibin\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d764999-f70b-4528-a03e-6e8eae6d6bae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-multus-conf-dir\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-run-netns\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-cni-netd\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.766486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-etc-kubernetes\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4991788b-de70-4f49-87f4-cad8cabd7fe3-agent-certs\") pod \"konnectivity-agent-nh5dl\" (UID: \"4991788b-de70-4f49-87f4-cad8cabd7fe3\") " pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-run-multus-certs\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766404    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-sysconfig\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-host\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-kubelet\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-systemd-units\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/250e9a05-21d4-45d5-9113-f4594095ffdb-env-overrides\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-run-netns\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-cni-netd\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/250e9a05-21d4-45d5-9113-f4594095ffdb-ovnkube-script-lib\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-run-multus-certs\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.767189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-registration-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-host\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-systemd-units\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-sysconfig\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dmd\" (UniqueName: \"kubernetes.io/projected/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-kube-api-access-m9dmd\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-kubelet\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-cnibin\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d764999-f70b-4528-a03e-6e8eae6d6bae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fe991f9-27b4-4d56-a779-014879464f58-dbus\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766777    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/250e9a05-21d4-45d5-9113-f4594095ffdb-ovnkube-config\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d764999-f70b-4528-a03e-6e8eae6d6bae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fe991f9-27b4-4d56-a779-014879464f58-dbus\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-sys-fs\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/250e9a05-21d4-45d5-9113-f4594095ffdb-env-overrides\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.767800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766935    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-cnibin\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-var-lib-cni-multus\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.766986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-run-systemd\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767027    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-cnibin\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767038    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-var-lib-cni-multus\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-os-release\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.767047    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-modprobe-d\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-run-systemd\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767097    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-os-release\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-etc-openvswitch\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.767130    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs podName:52e1d619-e463-49f5-b06c-f55c9cb8d1f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.267110266 +0000 UTC m=+1.985373029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs") pod "network-metrics-daemon-jn9vx" (UID: "52e1d619-e463-49f5-b06c-f55c9cb8d1f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-etc-openvswitch\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.768540 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-modprobe-d\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767166    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d764999-f70b-4528-a03e-6e8eae6d6bae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-system-cni-dir\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-system-cni-dir\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-var-lib-cni-bin\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-host-var-lib-cni-bin\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767265    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-systemd\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-run\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-tmp\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-systemd\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-slash\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:41.769271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfpnp\" (UniqueName: \"kubernetes.io/projected/fad79db2-2b92-4a09-8cec-ca7cb742b473-kube-api-access-jfpnp\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-run\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-run-ovn\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-slash\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-etc-selinux\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36eb1c86-7239-4520-bdf2-aaac34595c7d-iptables-alerter-script\") pod \"iptables-alerter-lzh6n\" (UID: \"36eb1c86-7239-4520-bdf2-aaac34595c7d\") " pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-run-ovn\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.767439    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36eb1c86-7239-4520-bdf2-aaac34595c7d-host-slash\") pod \"iptables-alerter-lzh6n\" (UID: \"36eb1c86-7239-4520-bdf2-aaac34595c7d\") " pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.767502    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret podName:0fe991f9-27b4-4d56-a779-014879464f58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.267489537 +0000 UTC m=+1.985752282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret") pod "global-pull-secret-syncer-lznbs" (UID: "0fe991f9-27b4-4d56-a779-014879464f58") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d764999-f70b-4528-a03e-6e8eae6d6bae-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-sys\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/250e9a05-21d4-45d5-9113-f4594095ffdb-ovn-node-metrics-cert\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.769807 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4991788b-de70-4f49-87f4-cad8cabd7fe3-konnectivity-ca\") pod \"konnectivity-agent-nh5dl\" (UID: \"4991788b-de70-4f49-87f4-cad8cabd7fe3\") " pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/364ccd55-cb96-46e5-a706-bcdf005bf715-serviceca\") pod \"node-ca-2ptrg\" (UID: \"364ccd55-cb96-46e5-a706-bcdf005bf715\") " pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-sys\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vcx\" (UniqueName: \"kubernetes.io/projected/364ccd55-cb96-46e5-a706-bcdf005bf715-kube-api-access-d7vcx\") pod \"node-ca-2ptrg\" (UID: \"364ccd55-cb96-46e5-a706-bcdf005bf715\") " pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767736    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/506b3c80-5b84-408c-b9ed-55b660477968-multus-daemon-config\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-lib-modules\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-tuned\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-run-openvswitch\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-os-release\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-hostroot\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p6fb\" (UniqueName: \"kubernetes.io/projected/506b3c80-5b84-408c-b9ed-55b660477968-kube-api-access-2p6fb\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-node-log\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d764999-f70b-4528-a03e-6e8eae6d6bae-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-log-socket\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.767978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhpg\" (UniqueName: \"kubernetes.io/projected/250e9a05-21d4-45d5-9113-f4594095ffdb-kube-api-access-xmhpg\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.770418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr75c\" (UniqueName: \"kubernetes.io/projected/36eb1c86-7239-4520-bdf2-aaac34595c7d-kube-api-access-tr75c\") pod \"iptables-alerter-lzh6n\" (UID: \"36eb1c86-7239-4520-bdf2-aaac34595c7d\") " pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-run-openvswitch\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-multus-cni-dir\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-kubernetes\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-sysctl-conf\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d764999-f70b-4528-a03e-6e8eae6d6bae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-lib-modules\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-sysctl-conf\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-var-lib-kubelet\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768273    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768291    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-hostroot\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/364ccd55-cb96-46e5-a706-bcdf005bf715-host\") pod \"node-ca-2ptrg\" (UID: \"364ccd55-cb96-46e5-a706-bcdf005bf715\") " pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-multus-socket-dir-parent\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-multus-socket-dir-parent\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-node-log\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.770899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-log-socket\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4991788b-de70-4f49-87f4-cad8cabd7fe3-konnectivity-ca\") pod \"konnectivity-agent-nh5dl\" (UID: \"4991788b-de70-4f49-87f4-cad8cabd7fe3\") " pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-multus-cni-dir\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/506b3c80-5b84-408c-b9ed-55b660477968-multus-daemon-config\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-kubernetes\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.768899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-var-lib-kubelet\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.769010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/250e9a05-21d4-45d5-9113-f4594095ffdb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.769079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/506b3c80-5b84-408c-b9ed-55b660477968-os-release\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.769126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/364ccd55-cb96-46e5-a706-bcdf005bf715-host\") pod \"node-ca-2ptrg\" (UID: \"364ccd55-cb96-46e5-a706-bcdf005bf715\") " pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.769151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/364ccd55-cb96-46e5-a706-bcdf005bf715-serviceca\") pod \"node-ca-2ptrg\" (UID: \"364ccd55-cb96-46e5-a706-bcdf005bf715\") " pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.770248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-tmp\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.770330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/250e9a05-21d4-45d5-9113-f4594095ffdb-ovn-node-metrics-cert\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.770391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-etc-tuned\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.771347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.770448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4991788b-de70-4f49-87f4-cad8cabd7fe3-agent-certs\") pod \"konnectivity-agent-nh5dl\" (UID: \"4991788b-de70-4f49-87f4-cad8cabd7fe3\") " pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:04:41.817803 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.817719    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:41.817803 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.817760    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:41.817803 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.817775    2565 projected.go:196] Error preparing data for projected volume kube-api-access-vfh5n for pod openshift-network-diagnostics/network-check-target-h2cxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:41.818072 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:41.817843    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n podName:ec04d48c-c71a-4abd-85be-a98ff75e17a1 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.317823936 +0000 UTC m=+2.036086681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vfh5n" (UniqueName: "kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n") pod "network-check-target-h2cxx" (UID: "ec04d48c-c71a-4abd-85be-a98ff75e17a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:41.820438 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.820409    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttzd\" (UniqueName: \"kubernetes.io/projected/0ba3f626-debc-4c53-8b0f-a10ef0411fbd-kube-api-access-zttzd\") pod \"tuned-p2sr6\" (UID: \"0ba3f626-debc-4c53-8b0f-a10ef0411fbd\") " pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:41.820548 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.820465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vcx\" (UniqueName: \"kubernetes.io/projected/364ccd55-cb96-46e5-a706-bcdf005bf715-kube-api-access-d7vcx\") pod \"node-ca-2ptrg\" (UID: \"364ccd55-cb96-46e5-a706-bcdf005bf715\") " pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.820548 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.820512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlwtc\" (UniqueName: \"kubernetes.io/projected/3c7b15ac-cf17-4a04-be09-eaadda984155-kube-api-access-rlwtc\") pod \"node-resolver-4s6pg\" (UID: \"3c7b15ac-cf17-4a04-be09-eaadda984155\") " pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:41.820633 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.820591    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p6fb\" (UniqueName: \"kubernetes.io/projected/506b3c80-5b84-408c-b9ed-55b660477968-kube-api-access-2p6fb\") pod \"multus-4kfcs\" (UID: \"506b3c80-5b84-408c-b9ed-55b660477968\") " pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.821013 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.820989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjgh\" (UniqueName: \"kubernetes.io/projected/7d764999-f70b-4528-a03e-6e8eae6d6bae-kube-api-access-9wjgh\") pod \"multus-additional-cni-plugins-9lq7r\" (UID: \"7d764999-f70b-4528-a03e-6e8eae6d6bae\") " pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:41.821163 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.821145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dmd\" (UniqueName: \"kubernetes.io/projected/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-kube-api-access-m9dmd\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:41.821163 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.821157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhpg\" (UniqueName: \"kubernetes.io/projected/250e9a05-21d4-45d5-9113-f4594095ffdb-kube-api-access-xmhpg\") pod \"ovnkube-node-dlqjl\" (UID: \"250e9a05-21d4-45d5-9113-f4594095ffdb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:41.848764 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.848738    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtkhm"
Jul 09 18:04:41.863477 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:41.863440    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7423fcc2f22fef5c1cd5088939f63a04.slice/crio-82f9afe5b0d90a4df588de2fbf6077b55607315613e67a6c91b96cb4a1cd0eeb WatchSource:0}: Error finding container 82f9afe5b0d90a4df588de2fbf6077b55607315613e67a6c91b96cb4a1cd0eeb: Status 404 returned error can't find the container with id 82f9afe5b0d90a4df588de2fbf6077b55607315613e67a6c91b96cb4a1cd0eeb
Jul 09 18:04:41.865095 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.865073    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtkhm"
Jul 09 18:04:41.868556 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868628 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868564    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-registration-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868628 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-sys-fs\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868729 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfpnp\" (UniqueName: \"kubernetes.io/projected/fad79db2-2b92-4a09-8cec-ca7cb742b473-kube-api-access-jfpnp\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868729 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-registration-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868729 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-etc-selinux\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868729 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868702    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-sys-fs\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868729 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36eb1c86-7239-4520-bdf2-aaac34595c7d-iptables-alerter-script\") pod \"iptables-alerter-lzh6n\" (UID: \"36eb1c86-7239-4520-bdf2-aaac34595c7d\") " pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.868986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36eb1c86-7239-4520-bdf2-aaac34595c7d-host-slash\") pod \"iptables-alerter-lzh6n\" (UID: \"36eb1c86-7239-4520-bdf2-aaac34595c7d\") " pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.868986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36eb1c86-7239-4520-bdf2-aaac34595c7d-host-slash\") pod \"iptables-alerter-lzh6n\" (UID: \"36eb1c86-7239-4520-bdf2-aaac34595c7d\") " pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.868986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868832    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-etc-selinux\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr75c\" (UniqueName: \"kubernetes.io/projected/36eb1c86-7239-4520-bdf2-aaac34595c7d-kube-api-access-tr75c\") pod \"iptables-alerter-lzh6n\" (UID: \"36eb1c86-7239-4520-bdf2-aaac34595c7d\") " pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.868986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-device-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.868986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.868984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-socket-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.869293 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.869128    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-socket-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.869293 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.869184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fad79db2-2b92-4a09-8cec-ca7cb742b473-device-dir\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.869293 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.869220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36eb1c86-7239-4520-bdf2-aaac34595c7d-iptables-alerter-script\") pod \"iptables-alerter-lzh6n\" (UID: \"36eb1c86-7239-4520-bdf2-aaac34595c7d\") " pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.870650 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.870634    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:04:41.870849 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:41.870831    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ab68799b6023da576fe8586ec62200.slice/crio-4f9cebeaf2baf09678a534080ed17d6897ceebb4ec8fc6f52309b8b8b88f5ddf WatchSource:0}: Error finding container 4f9cebeaf2baf09678a534080ed17d6897ceebb4ec8fc6f52309b8b8b88f5ddf: Status 404 returned error can't find the container with id 4f9cebeaf2baf09678a534080ed17d6897ceebb4ec8fc6f52309b8b8b88f5ddf
Jul 09 18:04:41.886462 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.886442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfpnp\" (UniqueName: \"kubernetes.io/projected/fad79db2-2b92-4a09-8cec-ca7cb742b473-kube-api-access-jfpnp\") pod \"aws-ebs-csi-driver-node-w8cc6\" (UID: \"fad79db2-2b92-4a09-8cec-ca7cb742b473\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:41.890483 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.890465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr75c\" (UniqueName: \"kubernetes.io/projected/36eb1c86-7239-4520-bdf2-aaac34595c7d-kube-api-access-tr75c\") pod \"iptables-alerter-lzh6n\" (UID: \"36eb1c86-7239-4520-bdf2-aaac34595c7d\") " pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:41.981455 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.981424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2ptrg"
Jul 09 18:04:41.986447 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:41.986422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4kfcs"
Jul 09 18:04:41.989816 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:41.989793    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364ccd55_cb96_46e5_a706_bcdf005bf715.slice/crio-34e8aeb76bb19e0927081809f1b32eaca4dfb894dc225f151f48bddd244bc491 WatchSource:0}: Error finding container 34e8aeb76bb19e0927081809f1b32eaca4dfb894dc225f151f48bddd244bc491: Status 404 returned error can't find the container with id 34e8aeb76bb19e0927081809f1b32eaca4dfb894dc225f151f48bddd244bc491
Jul 09 18:04:41.994633 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:41.994612    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506b3c80_5b84_408c_b9ed_55b660477968.slice/crio-186a5f688ac6b938ae8e520440857e1d249c6db4920e45affaeefb60371517f9 WatchSource:0}: Error finding container 186a5f688ac6b938ae8e520440857e1d249c6db4920e45affaeefb60371517f9: Status 404 returned error can't find the container with id 186a5f688ac6b938ae8e520440857e1d249c6db4920e45affaeefb60371517f9
Jul 09 18:04:42.000822 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.000802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9lq7r"
Jul 09 18:04:42.007943 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:42.007921    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d764999_f70b_4528_a03e_6e8eae6d6bae.slice/crio-ba8f62add17e5a6f5cd9e17256e3757439f9b0035d3f208b473dcdc57352f78a WatchSource:0}: Error finding container ba8f62add17e5a6f5cd9e17256e3757439f9b0035d3f208b473dcdc57352f78a: Status 404 returned error can't find the container with id ba8f62add17e5a6f5cd9e17256e3757439f9b0035d3f208b473dcdc57352f78a
Jul 09 18:04:42.023893 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.023868    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:04:42.031274 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:42.031247    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250e9a05_21d4_45d5_9113_f4594095ffdb.slice/crio-0fab36a42af5501db0fbde641935866dc6328dcb09077ca11907f9abd9ee1b90 WatchSource:0}: Error finding container 0fab36a42af5501db0fbde641935866dc6328dcb09077ca11907f9abd9ee1b90: Status 404 returned error can't find the container with id 0fab36a42af5501db0fbde641935866dc6328dcb09077ca11907f9abd9ee1b90
Jul 09 18:04:42.036634 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.036614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p2sr6"
Jul 09 18:04:42.043737 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:42.043712    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba3f626_debc_4c53_8b0f_a10ef0411fbd.slice/crio-90bccd256f2a7e274702e94179b550149eeba261431dcdeb1691449df5935dd3 WatchSource:0}: Error finding container 90bccd256f2a7e274702e94179b550149eeba261431dcdeb1691449df5935dd3: Status 404 returned error can't find the container with id 90bccd256f2a7e274702e94179b550149eeba261431dcdeb1691449df5935dd3
Jul 09 18:04:42.056157 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.056130    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:04:42.061865 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.061848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4s6pg"
Jul 09 18:04:42.063129 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:42.063106    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4991788b_de70_4f49_87f4_cad8cabd7fe3.slice/crio-67b046ba4166f4afcdd12c1a455ed1baf48b6cbf8790b4565029602989f26570 WatchSource:0}: Error finding container 67b046ba4166f4afcdd12c1a455ed1baf48b6cbf8790b4565029602989f26570: Status 404 returned error can't find the container with id 67b046ba4166f4afcdd12c1a455ed1baf48b6cbf8790b4565029602989f26570
Jul 09 18:04:42.069086 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:42.069031    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7b15ac_cf17_4a04_be09_eaadda984155.slice/crio-5e49c6fc53786b3b82c4104e02145264df62232b7f9c0c3a8d56351d25964ff3 WatchSource:0}: Error finding container 5e49c6fc53786b3b82c4104e02145264df62232b7f9c0c3a8d56351d25964ff3: Status 404 returned error can't find the container with id 5e49c6fc53786b3b82c4104e02145264df62232b7f9c0c3a8d56351d25964ff3
Jul 09 18:04:42.079655 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.079633    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6"
Jul 09 18:04:42.085371 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.085351    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lzh6n"
Jul 09 18:04:42.087622 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:42.087601    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad79db2_2b92_4a09_8cec_ca7cb742b473.slice/crio-28252ef99cddb164ebbab91c34f597969afaf4b053a638206b501751fb73a2d1 WatchSource:0}: Error finding container 28252ef99cddb164ebbab91c34f597969afaf4b053a638206b501751fb73a2d1: Status 404 returned error can't find the container with id 28252ef99cddb164ebbab91c34f597969afaf4b053a638206b501751fb73a2d1
Jul 09 18:04:42.092390 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:04:42.092371    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36eb1c86_7239_4520_bdf2_aaac34595c7d.slice/crio-e74d5b6c579ecf4603e7d3efa79ca5e3ad0274ab80ea4ee3bccf7cdbcdd485bb WatchSource:0}: Error finding container e74d5b6c579ecf4603e7d3efa79ca5e3ad0274ab80ea4ee3bccf7cdbcdd485bb: Status 404 returned error can't find the container with id e74d5b6c579ecf4603e7d3efa79ca5e3ad0274ab80ea4ee3bccf7cdbcdd485bb
Jul 09 18:04:42.168348 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.168321    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:42.270841 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.270803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:42.271044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.270848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:42.271044 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:42.270976    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:42.271044 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:42.271037    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret podName:0fe991f9-27b4-4d56-a779-014879464f58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:43.27102032 +0000 UTC m=+2.989283080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret") pod "global-pull-secret-syncer-lznbs" (UID: "0fe991f9-27b4-4d56-a779-014879464f58") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:42.271207 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:42.270976    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:42.271207 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:42.271108    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs podName:52e1d619-e463-49f5-b06c-f55c9cb8d1f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:43.271096015 +0000 UTC m=+2.989358761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs") pod "network-metrics-daemon-jn9vx" (UID: "52e1d619-e463-49f5-b06c-f55c9cb8d1f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:42.372091 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.372005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:42.372239 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:42.372190    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:42.372239 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:42.372218    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:42.372239 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:42.372229    2565 projected.go:196] Error preparing data for projected volume kube-api-access-vfh5n for pod openshift-network-diagnostics/network-check-target-h2cxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:42.372381 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:42.372283    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n podName:ec04d48c-c71a-4abd-85be-a98ff75e17a1 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:43.372267199 +0000 UTC m=+3.090529941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vfh5n" (UniqueName: "kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n") pod "network-check-target-h2cxx" (UID: "ec04d48c-c71a-4abd-85be-a98ff75e17a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:42.487674 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.487638    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:42.864844 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.864312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:42.864844 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:42.864444    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:42.869699 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.869617    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:41 +0000 UTC" deadline="2028-01-30 08:56:28.917783595 +0000 UTC"
Jul 09 18:04:42.869699 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.869652    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13670h51m46.048135648s"
Jul 09 18:04:42.893113 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.892986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p2sr6" event={"ID":"0ba3f626-debc-4c53-8b0f-a10ef0411fbd","Type":"ContainerStarted","Data":"90bccd256f2a7e274702e94179b550149eeba261431dcdeb1691449df5935dd3"}
Jul 09 18:04:42.897973 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.897942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerStarted","Data":"0fab36a42af5501db0fbde641935866dc6328dcb09077ca11907f9abd9ee1b90"}
Jul 09 18:04:42.900855 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.900763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2ptrg" event={"ID":"364ccd55-cb96-46e5-a706-bcdf005bf715","Type":"ContainerStarted","Data":"34e8aeb76bb19e0927081809f1b32eaca4dfb894dc225f151f48bddd244bc491"}
Jul 09 18:04:42.906595 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.906563    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal" event={"ID":"d1ab68799b6023da576fe8586ec62200","Type":"ContainerStarted","Data":"4f9cebeaf2baf09678a534080ed17d6897ceebb4ec8fc6f52309b8b8b88f5ddf"}
Jul 09 18:04:42.908415 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.908392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal" event={"ID":"7423fcc2f22fef5c1cd5088939f63a04","Type":"ContainerStarted","Data":"82f9afe5b0d90a4df588de2fbf6077b55607315613e67a6c91b96cb4a1cd0eeb"}
Jul 09 18:04:42.913347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.913322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6" event={"ID":"fad79db2-2b92-4a09-8cec-ca7cb742b473","Type":"ContainerStarted","Data":"28252ef99cddb164ebbab91c34f597969afaf4b053a638206b501751fb73a2d1"}
Jul 09 18:04:42.922153 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.922127    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4s6pg" event={"ID":"3c7b15ac-cf17-4a04-be09-eaadda984155","Type":"ContainerStarted","Data":"5e49c6fc53786b3b82c4104e02145264df62232b7f9c0c3a8d56351d25964ff3"}
Jul 09 18:04:42.934080 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.934048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lq7r" event={"ID":"7d764999-f70b-4528-a03e-6e8eae6d6bae","Type":"ContainerStarted","Data":"ba8f62add17e5a6f5cd9e17256e3757439f9b0035d3f208b473dcdc57352f78a"}
Jul 09 18:04:42.938570 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.938541    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4kfcs" event={"ID":"506b3c80-5b84-408c-b9ed-55b660477968","Type":"ContainerStarted","Data":"186a5f688ac6b938ae8e520440857e1d249c6db4920e45affaeefb60371517f9"}
Jul 09 18:04:42.944429 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.944400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lzh6n" event={"ID":"36eb1c86-7239-4520-bdf2-aaac34595c7d","Type":"ContainerStarted","Data":"e74d5b6c579ecf4603e7d3efa79ca5e3ad0274ab80ea4ee3bccf7cdbcdd485bb"}
Jul 09 18:04:42.948109 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:42.948081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nh5dl" event={"ID":"4991788b-de70-4f49-87f4-cad8cabd7fe3","Type":"ContainerStarted","Data":"67b046ba4166f4afcdd12c1a455ed1baf48b6cbf8790b4565029602989f26570"}
Jul 09 18:04:43.280131 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:43.280021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:43.280131 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:43.280078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:43.280368 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.280286    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:43.280368 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.280350    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs podName:52e1d619-e463-49f5-b06c-f55c9cb8d1f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:45.280330546 +0000 UTC m=+4.998593290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs") pod "network-metrics-daemon-jn9vx" (UID: "52e1d619-e463-49f5-b06c-f55c9cb8d1f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:43.280722 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.280696    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:43.280816 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.280780    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret podName:0fe991f9-27b4-4d56-a779-014879464f58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:45.280763063 +0000 UTC m=+4.999025811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret") pod "global-pull-secret-syncer-lznbs" (UID: "0fe991f9-27b4-4d56-a779-014879464f58") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:43.363341 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:43.362517    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:43.380646 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:43.380604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:43.380856 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.380833    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:43.380856 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.380854    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:43.381042 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.380867    2565 projected.go:196] Error preparing data for projected volume kube-api-access-vfh5n for pod openshift-network-diagnostics/network-check-target-h2cxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:43.381042 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.380943    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n podName:ec04d48c-c71a-4abd-85be-a98ff75e17a1 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:45.380925336 +0000 UTC m=+5.099188080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vfh5n" (UniqueName: "kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n") pod "network-check-target-h2cxx" (UID: "ec04d48c-c71a-4abd-85be-a98ff75e17a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:43.862517 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:43.862482    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:43.862708 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.862616    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:04:43.863084 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:43.863064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:43.863197 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:43.863177    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:43.870694 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:43.870640    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:41 +0000 UTC" deadline="2028-03-15 01:08:09.729806813 +0000 UTC"
Jul 09 18:04:43.870694 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:43.870692    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14743h3m25.8591189s"
Jul 09 18:04:44.865302 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:44.865014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:44.865460 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:44.865369    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:45.298985 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:45.298783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:45.298985 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:45.298838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:45.298985 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.298978    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:45.299541 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.299040    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret podName:0fe991f9-27b4-4d56-a779-014879464f58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:49.299023273 +0000 UTC m=+9.017286018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret") pod "global-pull-secret-syncer-lznbs" (UID: "0fe991f9-27b4-4d56-a779-014879464f58") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:45.299541 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.299434    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:45.299541 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.299488    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs podName:52e1d619-e463-49f5-b06c-f55c9cb8d1f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:49.299471724 +0000 UTC m=+9.017734470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs") pod "network-metrics-daemon-jn9vx" (UID: "52e1d619-e463-49f5-b06c-f55c9cb8d1f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:45.399410 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:45.399372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:45.399581 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.399536    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:45.399581 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.399557    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:45.399581 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.399567    2565 projected.go:196] Error preparing data for projected volume kube-api-access-vfh5n for pod openshift-network-diagnostics/network-check-target-h2cxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:45.399750 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.399629    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n podName:ec04d48c-c71a-4abd-85be-a98ff75e17a1 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:49.399610695 +0000 UTC m=+9.117873444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vfh5n" (UniqueName: "kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n") pod "network-check-target-h2cxx" (UID: "ec04d48c-c71a-4abd-85be-a98ff75e17a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:45.862530 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:45.862499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:45.862703 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.862632    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:04:45.862790 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:45.862760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:45.862950 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:45.862884    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:46.862583 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:46.862492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:46.863063 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:46.862683    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:47.862561 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:47.862078    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:47.862561 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:47.862223    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:47.862561 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:47.862468    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:47.863152 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:47.862580    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:04:48.862780 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:48.862438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:48.862780 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:48.862594    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:49.335203 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:49.335112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:49.335203 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:49.335165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:49.335436 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.335334    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:49.335436 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.335384    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret podName:0fe991f9-27b4-4d56-a779-014879464f58 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:57.33537121 +0000 UTC m=+17.053633952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret") pod "global-pull-secret-syncer-lznbs" (UID: "0fe991f9-27b4-4d56-a779-014879464f58") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:49.335812 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.335661    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:49.335812 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.335730    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs podName:52e1d619-e463-49f5-b06c-f55c9cb8d1f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:57.335711226 +0000 UTC m=+17.053973974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs") pod "network-metrics-daemon-jn9vx" (UID: "52e1d619-e463-49f5-b06c-f55c9cb8d1f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:49.435995 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:49.435857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:49.436186 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.436053    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:49.436186 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.436079    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:49.436186 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.436094    2565 projected.go:196] Error preparing data for projected volume kube-api-access-vfh5n for pod openshift-network-diagnostics/network-check-target-h2cxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:49.436186 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.436156    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n podName:ec04d48c-c71a-4abd-85be-a98ff75e17a1 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:57.436136311 +0000 UTC m=+17.154399057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vfh5n" (UniqueName: "kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n") pod "network-check-target-h2cxx" (UID: "ec04d48c-c71a-4abd-85be-a98ff75e17a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:49.862869 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:49.862828    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:49.863327 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.862997    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:49.863400 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:49.863381    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:49.863480 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:49.863464    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:04:50.863895 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:50.863850    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:50.864413 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:50.864000    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:51.862115 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:51.862027    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:51.862115 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:51.862059    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:51.862336 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:51.862148    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:51.862336 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:51.862280    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:04:52.863035 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:52.862998    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:52.863568 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:52.863139    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:53.862230 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:53.862195    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:53.862393 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:53.862199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:53.862393 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:53.862309    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:04:53.862497 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:53.862414    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:54.862837 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:54.862798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:54.863246 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:54.862934    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:55.862715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:55.862647    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:55.862921 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:55.862661    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:55.862921 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:55.862802    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:55.862921 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:55.862875    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:04:56.862667 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:56.862633    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:56.862833 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:56.862771    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:57.399331 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:57.399288    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:57.399331 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:57.399339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:57.399794 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.399458    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:57.399794 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.399464    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:57.399794 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.399516    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret podName:0fe991f9-27b4-4d56-a779-014879464f58 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.399499789 +0000 UTC m=+33.117762534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret") pod "global-pull-secret-syncer-lznbs" (UID: "0fe991f9-27b4-4d56-a779-014879464f58") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:57.399794 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.399535    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs podName:52e1d619-e463-49f5-b06c-f55c9cb8d1f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.399525478 +0000 UTC m=+33.117788220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs") pod "network-metrics-daemon-jn9vx" (UID: "52e1d619-e463-49f5-b06c-f55c9cb8d1f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:57.500751 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:57.500710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:57.500959 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.500873    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:57.500959 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.500894    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:57.500959 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.500918    2565 projected.go:196] Error preparing data for projected volume kube-api-access-vfh5n for pod openshift-network-diagnostics/network-check-target-h2cxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:57.501111 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.500978    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n podName:ec04d48c-c71a-4abd-85be-a98ff75e17a1 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.500961155 +0000 UTC m=+33.219223898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vfh5n" (UniqueName: "kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n") pod "network-check-target-h2cxx" (UID: "ec04d48c-c71a-4abd-85be-a98ff75e17a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:57.862819 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:57.862787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:57.863009 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:57.862787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:57.863009 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.862922    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:57.863009 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:57.862991    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:04:58.865082 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:58.865055    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:04:58.865490 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:58.865170    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:04:59.862234 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:59.862192    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:04:59.862388 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:59.862323    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:04:59.862488 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:04:59.862192    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:04:59.862610 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:04:59.862575    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:00.865410 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:00.865380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:00.865892 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:00.865473    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:05:01.862061 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:01.862014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:01.862218 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:01.862127    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:05:01.862218 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:01.862181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:01.862287 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:01.862267    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:02.862809 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:02.862530    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:02.863393 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:02.862885    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:05:02.997340 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:02.997261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal" event={"ID":"d1ab68799b6023da576fe8586ec62200","Type":"ContainerStarted","Data":"d96d70f15c43a60db3f0f03c952fe5eafbad459b7a12c2fdf7a2bc6142808bfd"}
Jul 09 18:05:02.998873 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:02.998853    2565 generic.go:356] "Generic (PLEG): container finished" podID="7423fcc2f22fef5c1cd5088939f63a04" containerID="a5d71b6c25a8da584440ad9f2a8ca500a4bee2b914638359b2c9667c099559f4" exitCode=0
Jul 09 18:05:02.998992 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:02.998926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal" event={"ID":"7423fcc2f22fef5c1cd5088939f63a04","Type":"ContainerDied","Data":"a5d71b6c25a8da584440ad9f2a8ca500a4bee2b914638359b2c9667c099559f4"}
Jul 09 18:05:02.999104 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:02.999089    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"
Jul 09 18:05:03.000295 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.000270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6" event={"ID":"fad79db2-2b92-4a09-8cec-ca7cb742b473","Type":"ContainerStarted","Data":"8eca6489be50a74857f260f4698d12e1a3bc3c95df72309b383735a790064b12"}
Jul 09 18:05:03.001632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.001613    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4s6pg" event={"ID":"3c7b15ac-cf17-4a04-be09-eaadda984155","Type":"ContainerStarted","Data":"e5a91d03cdfab7fecd55534ba9e425642fe48fb2111882138162d1552911b27a"}
Jul 09 18:05:03.003096 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.003075    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d764999-f70b-4528-a03e-6e8eae6d6bae" containerID="0c48b7c85ae39624b7caa2b4722e9ce83b8510acb283fb23e12f499f800e78e0" exitCode=0
Jul 09 18:05:03.003192 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.003133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lq7r" event={"ID":"7d764999-f70b-4528-a03e-6e8eae6d6bae","Type":"ContainerDied","Data":"0c48b7c85ae39624b7caa2b4722e9ce83b8510acb283fb23e12f499f800e78e0"}
Jul 09 18:05:03.004599 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.004578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4kfcs" event={"ID":"506b3c80-5b84-408c-b9ed-55b660477968","Type":"ContainerStarted","Data":"3e782c8b87f67e2bceaefdf29e0df9066110f48762217f8cbf5155e4018bf8ea"}
Jul 09 18:05:03.005958 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.005936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nh5dl" event={"ID":"4991788b-de70-4f49-87f4-cad8cabd7fe3","Type":"ContainerStarted","Data":"e0ee109c243284fbd2d75ba30fbeba90a366660d0b1e75c0b56893a2b2d8c97b"}
Jul 09 18:05:03.007355 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.007331    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p2sr6" event={"ID":"0ba3f626-debc-4c53-8b0f-a10ef0411fbd","Type":"ContainerStarted","Data":"2b8652f9fc1932bf6ae5aa1bfae24736ffedb10e1aed9e0fbe2d28f29e6c0506"}
Jul 09 18:05:03.010047 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.010031    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:05:03.010411 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.010394    2565 generic.go:356] "Generic (PLEG): container finished" podID="250e9a05-21d4-45d5-9113-f4594095ffdb" containerID="0169054f657ff762ddd9f9181a571b5bd057052873decfda4cbdd650878c9b76" exitCode=1
Jul 09 18:05:03.010511 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.010484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerStarted","Data":"6c4ea25a25a230d0fee8eef975a80f691a80db720881dfcd09427549afbb7def"}
Jul 09 18:05:03.010587 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.010572    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerStarted","Data":"304cc0aaeb8d618b73f574069019eef783dc0715203d8544deed25602cafeafe"}
Jul 09 18:05:03.010643 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.010629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerStarted","Data":"f3492eba562d069a102ddd396696c0191eddc59ee6d6feba07f2e7816b303db0"}
Jul 09 18:05:03.010691 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.010680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerDied","Data":"0169054f657ff762ddd9f9181a571b5bd057052873decfda4cbdd650878c9b76"}
Jul 09 18:05:03.010747 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.010733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerStarted","Data":"e83f1dddc67be302e20dedc868b82474be1a57604219c65f8d8bdc281db1e655"}
Jul 09 18:05:03.011674 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.011656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2ptrg" event={"ID":"364ccd55-cb96-46e5-a706-bcdf005bf715","Type":"ContainerStarted","Data":"a34e333787a64a3aa7989515e033f811cb2ce84ee996147a7606b55d3edd9c98"}
Jul 09 18:05:03.017157 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.017140    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:05:03.017881 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.017866    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal"]
Jul 09 18:05:03.019673 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.019638    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal" podStartSLOduration=22.019627816 podStartE2EDuration="22.019627816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:03.019153386 +0000 UTC m=+22.737416161" watchObservedRunningTime="2026-07-09 18:05:03.019627816 +0000 UTC m=+22.737890558"
Jul 09 18:05:03.039133 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.039000    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p2sr6" podStartSLOduration=1.7872364859999998 podStartE2EDuration="22.038981681s" totalImagesPullingTime="20.251745195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:03.038786206 +0000 UTC m=+22.757048971" watchObservedRunningTime="2026-07-09 18:05:03.038981681 +0000 UTC m=+22.757244446"
Jul 09 18:05:03.078513 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.078469    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nh5dl" podStartSLOduration=1.84737544 podStartE2EDuration="22.07845077s" totalImagesPullingTime="20.23107533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:03.078025064 +0000 UTC m=+22.796287830" watchObservedRunningTime="2026-07-09 18:05:03.07845077 +0000 UTC m=+22.796713536"
Jul 09 18:05:03.097624 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.097565    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4s6pg" podStartSLOduration=1.872161712 podStartE2EDuration="22.097544388s" totalImagesPullingTime="20.225382676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:03.09586759 +0000 UTC m=+22.814130356" watchObservedRunningTime="2026-07-09 18:05:03.097544388 +0000 UTC m=+22.815807154"
Jul 09 18:05:03.134887 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.134841    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2ptrg" podStartSLOduration=2.88527601 podStartE2EDuration="23.134825554s" totalImagesPullingTime="20.249549544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:03.134471421 +0000 UTC m=+22.852734185" watchObservedRunningTime="2026-07-09 18:05:03.134825554 +0000 UTC m=+22.853088318"
Jul 09 18:05:03.201667 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.201618    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4kfcs" podStartSLOduration=2.709207014 podStartE2EDuration="23.201600874s" totalImagesPullingTime="20.49239386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:03.201114946 +0000 UTC m=+22.919377722" watchObservedRunningTime="2026-07-09 18:05:03.201600874 +0000 UTC m=+22.919863638"
Jul 09 18:05:03.862974 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.862948    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:03.863531 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:03.863071    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:03.863531 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:03.862948    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:03.863531 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:03.863361    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:05:04.017188 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.017102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal" event={"ID":"7423fcc2f22fef5c1cd5088939f63a04","Type":"ContainerStarted","Data":"a42f68ac9c8df8fd31db3542db14d4565e1108dc33ba25b86b3e014685b74a52"}
Jul 09 18:05:04.019211 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.018817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lzh6n" event={"ID":"36eb1c86-7239-4520-bdf2-aaac34595c7d","Type":"ContainerStarted","Data":"b5b2b70e9e170b46ec4e9a488bdc07c8cee36a0b6e0acb902909493534fecbd4"}
Jul 09 18:05:04.022967 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.022946    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:05:04.023747 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.023722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerStarted","Data":"615a914644a67465e3191206729828d456251c756714306aab31194dd5c20b50"}
Jul 09 18:05:04.038977 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.038920    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-87.ec2.internal" podStartSLOduration=1.038887516 podStartE2EDuration="1.038887516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:04.038839098 +0000 UTC m=+23.757101862" watchObservedRunningTime="2026-07-09 18:05:04.038887516 +0000 UTC m=+23.757150279"
Jul 09 18:05:04.747993 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.747948    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:05:04.750392 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.750304    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:05:04.747974555Z","UUID":"3a46dbf0-800b-4fab-bafa-6ab48bd92c5e","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:05:04.751924 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.751880    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:05:04.752009 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.751938    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:05:04.862534 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:04.862503    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:04.862714 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:04.862633    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:05:05.027988 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:05.027945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6" event={"ID":"fad79db2-2b92-4a09-8cec-ca7cb742b473","Type":"ContainerStarted","Data":"95338ac4b806da5f11a2ebf31b1816cdf29a05881b9255e0cfa1f014440affd3"}
Jul 09 18:05:05.862490 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:05.862310    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:05.862678 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:05.862310    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:05.862678 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:05.862575    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:05.862785 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:05.862703    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:05:06.033835 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:06.033806    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:05:06.034308 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:06.034237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerStarted","Data":"1ed41c375551f2400db856ad3b0a6e032b87549d46bd85c00d29d45077b5f1d8"}
Jul 09 18:05:06.863173 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:06.863124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:06.863381 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:06.863267    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:05:07.797424 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:07.797251    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:05:07.798135 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:07.797924    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:05:07.828964 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:07.828894    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lzh6n" podStartSLOduration=6.681352285 podStartE2EDuration="26.828862931s" totalImagesPullingTime="20.147510646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:04.056666485 +0000 UTC m=+23.774929250" watchObservedRunningTime="2026-07-09 18:05:07.828862931 +0000 UTC m=+27.547125695"
Jul 09 18:05:07.863001 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:07.862886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:07.863001 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:07.862940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:07.863210 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:07.863038    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:05:07.863210 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:07.863175    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:08.042594 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:08.042570    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:05:08.042991 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:08.042965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerStarted","Data":"43e764c48016f875b85b4d05820dd2dcf7ddbc9a7e7830cb6135892065ee954c"}
Jul 09 18:05:08.043183 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:08.043158    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:05:08.043410 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:08.043391    2565 scope.go:117] "RemoveContainer" containerID="0169054f657ff762ddd9f9181a571b5bd057052873decfda4cbdd650878c9b76"
Jul 09 18:05:08.044944 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:08.044918    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d764999-f70b-4528-a03e-6e8eae6d6bae" containerID="6e90c4868b99d8ba19e55fb71e15390c04a4b1c9f8b159e7431349ecb027f429" exitCode=0
Jul 09 18:05:08.045015 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:08.044994    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lq7r" event={"ID":"7d764999-f70b-4528-a03e-6e8eae6d6bae","Type":"ContainerDied","Data":"6e90c4868b99d8ba19e55fb71e15390c04a4b1c9f8b159e7431349ecb027f429"}
Jul 09 18:05:08.059390 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:08.059371    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:05:08.862258 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:08.862224    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:08.862787 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:08.862323    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:05:09.049551 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.049516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6" event={"ID":"fad79db2-2b92-4a09-8cec-ca7cb742b473","Type":"ContainerStarted","Data":"25e2da36e20cd4a8f335900ea49b38e6a35093c4db27e06855ea61a05a726901"}
Jul 09 18:05:09.052871 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.052850    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:05:09.053237 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.053212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" event={"ID":"250e9a05-21d4-45d5-9113-f4594095ffdb","Type":"ContainerStarted","Data":"40b270beb96571a97dd4ea1f0b905a72168616f48a76ef42caae363ea8419fff"}
Jul 09 18:05:09.053397 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.053382    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:05:09.053449 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.053437    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:05:09.069585 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.069564    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:05:09.133662 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.133565    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w8cc6" podStartSLOduration=2.66918743 podStartE2EDuration="28.133548838s" totalImagesPullingTime="25.464361408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:09.089224876 +0000 UTC m=+28.807487639" watchObservedRunningTime="2026-07-09 18:05:09.133548838 +0000 UTC m=+28.851811602"
Jul 09 18:05:09.134004 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.133974    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl" podStartSLOduration=8.828434738 podStartE2EDuration="29.133961682s" totalImagesPullingTime="20.305526944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:09.132524917 +0000 UTC m=+28.850787692" watchObservedRunningTime="2026-07-09 18:05:09.133961682 +0000 UTC m=+28.852224445"
Jul 09 18:05:09.247888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.247850    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:05:09.248157 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.247994    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 18:05:09.248443 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.248413    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nh5dl"
Jul 09 18:05:09.862728 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.862511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:09.863132 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.862511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:09.863132 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:09.862772    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:05:09.863132 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:09.862817    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:09.927333 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.927298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jn9vx"]
Jul 09 18:05:09.937818 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.937779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lznbs"]
Jul 09 18:05:09.937991 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.937917    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:09.938043 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:09.938003    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:05:09.938280 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:09.938260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h2cxx"]
Jul 09 18:05:10.057634 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:10.057603    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d764999-f70b-4528-a03e-6e8eae6d6bae" containerID="b954703e5d27cedba3d539e7ad54b6d8e11f094026ad3337cafe6bd2564675e0" exitCode=0
Jul 09 18:05:10.057814 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:10.057717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lq7r" event={"ID":"7d764999-f70b-4528-a03e-6e8eae6d6bae","Type":"ContainerDied","Data":"b954703e5d27cedba3d539e7ad54b6d8e11f094026ad3337cafe6bd2564675e0"}
Jul 09 18:05:10.058321 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:10.058299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:10.058458 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:10.058392    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:10.058458 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:10.058402    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:10.058565 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:10.058495    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:05:11.862800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:11.862761    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:11.863485 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:11.862820    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:11.863485 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:11.862947    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:05:11.863485 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:11.863048    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:11.863485 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:11.863142    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:11.863485 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:11.863212    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:05:12.064541 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:12.064511    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d764999-f70b-4528-a03e-6e8eae6d6bae" containerID="38a19f50000a4d0ace3275544a50dc60bb2f6f74fad948f06c1cb1c04d5debf4" exitCode=0
Jul 09 18:05:12.064685 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:12.064548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lq7r" event={"ID":"7d764999-f70b-4528-a03e-6e8eae6d6bae","Type":"ContainerDied","Data":"38a19f50000a4d0ace3275544a50dc60bb2f6f74fad948f06c1cb1c04d5debf4"}
Jul 09 18:05:13.415547 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:13.415418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:13.415547 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:13.415468    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:13.416317 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.415596    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:05:13.416317 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.415653    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret podName:0fe991f9-27b4-4d56-a779-014879464f58 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.415636531 +0000 UTC m=+65.133899275 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret") pod "global-pull-secret-syncer-lznbs" (UID: "0fe991f9-27b4-4d56-a779-014879464f58") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:05:13.416317 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.415595    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:05:13.416317 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.415873    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs podName:52e1d619-e463-49f5-b06c-f55c9cb8d1f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.415859151 +0000 UTC m=+65.134121898 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs") pod "network-metrics-daemon-jn9vx" (UID: "52e1d619-e463-49f5-b06c-f55c9cb8d1f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:05:13.516017 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:13.515991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:13.516186 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.516167    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:05:13.516239 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.516191    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:05:13.516239 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.516202    2565 projected.go:196] Error preparing data for projected volume kube-api-access-vfh5n for pod openshift-network-diagnostics/network-check-target-h2cxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:05:13.516321 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.516253    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n podName:ec04d48c-c71a-4abd-85be-a98ff75e17a1 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.516237524 +0000 UTC m=+65.234500267 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vfh5n" (UniqueName: "kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n") pod "network-check-target-h2cxx" (UID: "ec04d48c-c71a-4abd-85be-a98ff75e17a1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:05:13.862218 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:13.862184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:13.862397 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.862314    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:13.862397 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:13.862375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:13.862516 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.862494    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:05:13.862676 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:13.862643    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:13.862796 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:13.862776    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:05:15.862379 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:15.862184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:15.862814 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:15.862184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:15.862814 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:15.862475    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h2cxx" podUID="ec04d48c-c71a-4abd-85be-a98ff75e17a1"
Jul 09 18:05:15.862814 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:15.862207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:15.862814 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:15.862561    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jn9vx" podUID="52e1d619-e463-49f5-b06c-f55c9cb8d1f2"
Jul 09 18:05:15.862814 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:15.862600    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lznbs" podUID="0fe991f9-27b4-4d56-a779-014879464f58"
Jul 09 18:05:16.060086 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.060052    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-87.ec2.internal" event="NodeReady"
Jul 09 18:05:16.060266 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.060215    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:05:16.147385 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.147275    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8l287"]
Jul 09 18:05:16.162953 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.162926    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vvfdf"]
Jul 09 18:05:16.163089 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.162995    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8l287"
Jul 09 18:05:16.166163 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.166140    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bf4m7\""
Jul 09 18:05:16.166412 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.166206    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:05:16.166412 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.166213    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:05:16.166534 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.166505    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:05:16.185229 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.185203    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8l287"]
Jul 09 18:05:16.185323 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.185235    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vvfdf"]
Jul 09 18:05:16.185368 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.185325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.188251 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.188236    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:05:16.188721 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.188706    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nq7m5\""
Jul 09 18:05:16.188811 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.188728    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:05:16.236340 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.236308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zxg\" (UniqueName: \"kubernetes.io/projected/e0a920b9-1318-44a0-8026-6f4a0defd42f-kube-api-access-m7zxg\") pod \"ingress-canary-8l287\" (UID: \"e0a920b9-1318-44a0-8026-6f4a0defd42f\") " pod="openshift-ingress-canary/ingress-canary-8l287"
Jul 09 18:05:16.236489 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.236409    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0a920b9-1318-44a0-8026-6f4a0defd42f-cert\") pod \"ingress-canary-8l287\" (UID: \"e0a920b9-1318-44a0-8026-6f4a0defd42f\") " pod="openshift-ingress-canary/ingress-canary-8l287"
Jul 09 18:05:16.337694 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.337651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0a920b9-1318-44a0-8026-6f4a0defd42f-cert\") pod \"ingress-canary-8l287\" (UID: \"e0a920b9-1318-44a0-8026-6f4a0defd42f\") " pod="openshift-ingress-canary/ingress-canary-8l287"
Jul 09 18:05:16.337694 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.337699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3656bece-edae-4e6d-b42a-7adca0774519-config-volume\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.337889 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.337744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.337889 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.337762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3656bece-edae-4e6d-b42a-7adca0774519-tmp-dir\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.337889 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.337802    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rdtt\" (UniqueName: \"kubernetes.io/projected/3656bece-edae-4e6d-b42a-7adca0774519-kube-api-access-4rdtt\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.337889 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.337870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zxg\" (UniqueName: \"kubernetes.io/projected/e0a920b9-1318-44a0-8026-6f4a0defd42f-kube-api-access-m7zxg\") pod \"ingress-canary-8l287\" (UID: \"e0a920b9-1318-44a0-8026-6f4a0defd42f\") " pod="openshift-ingress-canary/ingress-canary-8l287"
Jul 09 18:05:16.342399 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.342374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0a920b9-1318-44a0-8026-6f4a0defd42f-cert\") pod \"ingress-canary-8l287\" (UID: \"e0a920b9-1318-44a0-8026-6f4a0defd42f\") " pod="openshift-ingress-canary/ingress-canary-8l287"
Jul 09 18:05:16.349543 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.349515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zxg\" (UniqueName: \"kubernetes.io/projected/e0a920b9-1318-44a0-8026-6f4a0defd42f-kube-api-access-m7zxg\") pod \"ingress-canary-8l287\" (UID: \"e0a920b9-1318-44a0-8026-6f4a0defd42f\") " pod="openshift-ingress-canary/ingress-canary-8l287"
Jul 09 18:05:16.438432 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.438406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rdtt\" (UniqueName: \"kubernetes.io/projected/3656bece-edae-4e6d-b42a-7adca0774519-kube-api-access-4rdtt\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.438562 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.438478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3656bece-edae-4e6d-b42a-7adca0774519-config-volume\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.438562 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.438547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.438678 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.438574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3656bece-edae-4e6d-b42a-7adca0774519-tmp-dir\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.445317 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:16.445287    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:16.445439 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:16.445374    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls podName:3656bece-edae-4e6d-b42a-7adca0774519 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:16.9453529 +0000 UTC m=+36.663615645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls") pod "dns-default-vvfdf" (UID: "3656bece-edae-4e6d-b42a-7adca0774519") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:16.445715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.445677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3656bece-edae-4e6d-b42a-7adca0774519-tmp-dir\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.445862 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.445746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3656bece-edae-4e6d-b42a-7adca0774519-config-volume\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.454402 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.454380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rdtt\" (UniqueName: \"kubernetes.io/projected/3656bece-edae-4e6d-b42a-7adca0774519-kube-api-access-4rdtt\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.473535 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.473499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8l287"
Jul 09 18:05:16.629681 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.629221    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8l287"]
Jul 09 18:05:16.684780 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:05:16.684713    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a920b9_1318_44a0_8026_6f4a0defd42f.slice/crio-d0d79bdcd3a17ac249f9f930d72f790fdf43407db431c5072b2e66cad0d57f6c WatchSource:0}: Error finding container d0d79bdcd3a17ac249f9f930d72f790fdf43407db431c5072b2e66cad0d57f6c: Status 404 returned error can't find the container with id d0d79bdcd3a17ac249f9f930d72f790fdf43407db431c5072b2e66cad0d57f6c
Jul 09 18:05:16.948893 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:16.948861    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:16.949272 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:16.949037    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:16.949272 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:16.949105    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls podName:3656bece-edae-4e6d-b42a-7adca0774519 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:17.9490888 +0000 UTC m=+37.667351542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls") pod "dns-default-vvfdf" (UID: "3656bece-edae-4e6d-b42a-7adca0774519") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:17.077560 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.077521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8l287" event={"ID":"e0a920b9-1318-44a0-8026-6f4a0defd42f","Type":"ContainerStarted","Data":"d0d79bdcd3a17ac249f9f930d72f790fdf43407db431c5072b2e66cad0d57f6c"}
Jul 09 18:05:17.080283 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.080255    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d764999-f70b-4528-a03e-6e8eae6d6bae" containerID="d3bc0b40480fab51ef29d08946c5f5ebb0aad5153520f223a7053126c68aef2c" exitCode=0
Jul 09 18:05:17.080420 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.080294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lq7r" event={"ID":"7d764999-f70b-4528-a03e-6e8eae6d6bae","Type":"ContainerDied","Data":"d3bc0b40480fab51ef29d08946c5f5ebb0aad5153520f223a7053126c68aef2c"}
Jul 09 18:05:17.862716 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.862510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:17.862889 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.862543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:17.862889 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.862543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:17.868392 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.868348    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:17.868547 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.868474    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:17.868760 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.868743    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcghf\""
Jul 09 18:05:17.869140 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.869101    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:17.869140 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.869130    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:17.871631 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.871567    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhwtc\""
Jul 09 18:05:17.957263 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:17.957230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:17.957622 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:17.957387    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:17.957622 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:17.957456    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls podName:3656bece-edae-4e6d-b42a-7adca0774519 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:19.957440944 +0000 UTC m=+39.675703691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls") pod "dns-default-vvfdf" (UID: "3656bece-edae-4e6d-b42a-7adca0774519") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:18.086631 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:18.086596    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d764999-f70b-4528-a03e-6e8eae6d6bae" containerID="f8417fa2f12c768ac45e995fe6c6d4d4dc5ccd7383e5939754f277165177b389" exitCode=0
Jul 09 18:05:18.086793 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:18.086640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lq7r" event={"ID":"7d764999-f70b-4528-a03e-6e8eae6d6bae","Type":"ContainerDied","Data":"f8417fa2f12c768ac45e995fe6c6d4d4dc5ccd7383e5939754f277165177b389"}
Jul 09 18:05:19.092328 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:19.092234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lq7r" event={"ID":"7d764999-f70b-4528-a03e-6e8eae6d6bae","Type":"ContainerStarted","Data":"c3e9697ffe07685e7e67988221b9fe2b5a937446f5487a9db4e31ceed7d19228"}
Jul 09 18:05:19.093534 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:19.093509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8l287" event={"ID":"e0a920b9-1318-44a0-8026-6f4a0defd42f","Type":"ContainerStarted","Data":"54ffa2f212b929e6eeb178ced177e8667a2eb8dbad7a8c699c949f89474ef11f"}
Jul 09 18:05:19.126014 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:19.125932    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9lq7r" podStartSLOduration=7.189617805 podStartE2EDuration="39.125900391s" totalImagesPullingTime="31.936282586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:19.12459637 +0000 UTC m=+38.842859135" watchObservedRunningTime="2026-07-09 18:05:19.125900391 +0000 UTC m=+38.844163156"
Jul 09 18:05:19.150452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:19.150403    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8l287" podStartSLOduration=1.084488683 podStartE2EDuration="3.150385058s" totalImagesPullingTime="2.065896375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:19.149720964 +0000 UTC m=+38.867983728" watchObservedRunningTime="2026-07-09 18:05:19.150385058 +0000 UTC m=+38.868647823"
Jul 09 18:05:19.971561 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:19.971522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:19.971724 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:19.971632    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:19.971724 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:19.971684    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls podName:3656bece-edae-4e6d-b42a-7adca0774519 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:23.971670166 +0000 UTC m=+43.689932909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls") pod "dns-default-vvfdf" (UID: "3656bece-edae-4e6d-b42a-7adca0774519") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:23.998611 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:23.998567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:23.999126 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:23.998688    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:23.999126 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:23.998743    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls podName:3656bece-edae-4e6d-b42a-7adca0774519 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:31.998729331 +0000 UTC m=+51.716992072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls") pod "dns-default-vvfdf" (UID: "3656bece-edae-4e6d-b42a-7adca0774519") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:29.548856 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.548825    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ggcq4"]
Jul 09 18:05:29.594060 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.594026    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ggcq4"]
Jul 09 18:05:29.594199 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.594151    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.597177 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.597142    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:05:29.597303 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.597227    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:05:29.598210 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.598192    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qh478\""
Jul 09 18:05:29.598270 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.598220    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:05:29.598516 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.598501    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:05:29.639246 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.639201    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq"]
Jul 09 18:05:29.651761 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.651732    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq"
Jul 09 18:05:29.654688 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.654664    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq"]
Jul 09 18:05:29.654975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.654961    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 18:05:29.655409 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.655387    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7bqcw\""
Jul 09 18:05:29.655642 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.655621    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 18:05:29.741408 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.741374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqzq\" (UniqueName: \"kubernetes.io/projected/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-api-access-4rqzq\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.741408 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.741422    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.741640 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.741449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53d9f28c-f462-41ba-a0e8-b376d1d7a962-data-volume\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.741640 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.741471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.741640 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.741512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53d9f28c-f462-41ba-a0e8-b376d1d7a962-crio-socket\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.841932 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.841831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53d9f28c-f462-41ba-a0e8-b376d1d7a962-crio-socket\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.841932 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.841897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbt4\" (UniqueName: \"kubernetes.io/projected/1e138bfa-74f8-48c7-baa3-b233cd50bd4a-kube-api-access-8wbt4\") pod \"migrator-7f59b54f6c-z6jvq\" (UID: \"1e138bfa-74f8-48c7-baa3-b233cd50bd4a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq"
Jul 09 18:05:29.842157 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.841938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqzq\" (UniqueName: \"kubernetes.io/projected/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-api-access-4rqzq\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.842157 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.841966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.842157 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.841994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53d9f28c-f462-41ba-a0e8-b376d1d7a962-data-volume\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.842157 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.842051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.842157 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.842081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53d9f28c-f462-41ba-a0e8-b376d1d7a962-crio-socket\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.842157 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:29.842086    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:29.842401 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:29.842157    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:29.842401 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:29.842163    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:30.342143752 +0000 UTC m=+50.060406512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:29.842401 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:29.842235    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:30.342214481 +0000 UTC m=+50.060477230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:29.842401 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.842249    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53d9f28c-f462-41ba-a0e8-b376d1d7a962-data-volume\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.852962 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.852939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqzq\" (UniqueName: \"kubernetes.io/projected/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-api-access-4rqzq\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:29.943330 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.943297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbt4\" (UniqueName: \"kubernetes.io/projected/1e138bfa-74f8-48c7-baa3-b233cd50bd4a-kube-api-access-8wbt4\") pod \"migrator-7f59b54f6c-z6jvq\" (UID: \"1e138bfa-74f8-48c7-baa3-b233cd50bd4a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq"
Jul 09 18:05:29.953625 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.953600    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbt4\" (UniqueName: \"kubernetes.io/projected/1e138bfa-74f8-48c7-baa3-b233cd50bd4a-kube-api-access-8wbt4\") pod \"migrator-7f59b54f6c-z6jvq\" (UID: \"1e138bfa-74f8-48c7-baa3-b233cd50bd4a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq"
Jul 09 18:05:29.961506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:29.961484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq"
Jul 09 18:05:30.118232 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:30.118158    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq"]
Jul 09 18:05:30.121320 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:05:30.121296    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e138bfa_74f8_48c7_baa3_b233cd50bd4a.slice/crio-55d98191947fa713e8f11cb505ce921e1fadcd9f0bb8650709d49ca6601aaaa2 WatchSource:0}: Error finding container 55d98191947fa713e8f11cb505ce921e1fadcd9f0bb8650709d49ca6601aaaa2: Status 404 returned error can't find the container with id 55d98191947fa713e8f11cb505ce921e1fadcd9f0bb8650709d49ca6601aaaa2
Jul 09 18:05:30.346884 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:30.346844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:30.347061 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:30.346893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:30.347061 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:30.347013    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:30.347130 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:30.347074    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:31.347059732 +0000 UTC m=+51.065322474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:30.347130 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:30.347094    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:30.347204 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:30.347149    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:31.347134514 +0000 UTC m=+51.065397259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:31.124534 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:31.124501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq" event={"ID":"1e138bfa-74f8-48c7-baa3-b233cd50bd4a","Type":"ContainerStarted","Data":"55d98191947fa713e8f11cb505ce921e1fadcd9f0bb8650709d49ca6601aaaa2"}
Jul 09 18:05:31.355031 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:31.354977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:31.355031 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:31.355033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:31.355260 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:31.355120    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:31.355260 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:31.355198    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:33.355173352 +0000 UTC m=+53.073436097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:31.355260 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:31.355133    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:31.355260 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:31.355248    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:33.355236496 +0000 UTC m=+53.073499237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:32.060716 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:32.060614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:32.060885 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:32.060751    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:32.060885 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:32.060819    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls podName:3656bece-edae-4e6d-b42a-7adca0774519 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:48.060800403 +0000 UTC m=+67.779063145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls") pod "dns-default-vvfdf" (UID: "3656bece-edae-4e6d-b42a-7adca0774519") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:32.128557 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:32.128520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq" event={"ID":"1e138bfa-74f8-48c7-baa3-b233cd50bd4a","Type":"ContainerStarted","Data":"fa9ac4a2ec6fbe222cf068f99f4d631ad65872549d4fd24aaa705bd7367c2ec2"}
Jul 09 18:05:32.128927 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:32.128580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq" event={"ID":"1e138bfa-74f8-48c7-baa3-b233cd50bd4a","Type":"ContainerStarted","Data":"88758065079e0e0a457e3ec82a35fe541eb03cdfeb2a2c5b84fe667c5b03e845"}
Jul 09 18:05:32.192703 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:32.192679    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4s6pg_3c7b15ac-cf17-4a04-be09-eaadda984155/dns-node-resolver/0.log"
Jul 09 18:05:33.173974 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:33.173946    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2ptrg_364ccd55-cb96-46e5-a706-bcdf005bf715/node-ca/0.log"
Jul 09 18:05:33.374210 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:33.374171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:33.374403 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:33.374230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:33.374403 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:33.374300    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:33.374403 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:33.374336    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:33.374403 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:33.374376    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:37.374359125 +0000 UTC m=+57.092621869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:33.374403 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:33.374391    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:37.374385416 +0000 UTC m=+57.092648158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:33.972694 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:33.972668    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8l287_e0a920b9-1318-44a0-8026-6f4a0defd42f/serve-healthcheck-canary/0.log"
Jul 09 18:05:34.573715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:34.573676    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z6jvq_1e138bfa-74f8-48c7-baa3-b233cd50bd4a/migrator/0.log"
Jul 09 18:05:34.772447 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:34.772418    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z6jvq_1e138bfa-74f8-48c7-baa3-b233cd50bd4a/graceful-termination/0.log"
Jul 09 18:05:37.398245 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:37.398198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:37.398699 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:37.398338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:37.398699 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:37.398342    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:37.398699 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:37.398375    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:05:37.398699 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:37.398436    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.398420153 +0000 UTC m=+65.116682894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : configmap "kube-rbac-proxy" not found
Jul 09 18:05:37.398699 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:37.398452    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.398445562 +0000 UTC m=+65.116708304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:41.070018 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:41.069977    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dlqjl"
Jul 09 18:05:41.108645 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:41.108554    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6jvq" podStartSLOduration=10.526812276 podStartE2EDuration="12.108535481s" totalImagesPullingTime="1.581723205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:32.155605149 +0000 UTC m=+51.873867935" watchObservedRunningTime="2026-07-09 18:05:41.108535481 +0000 UTC m=+60.826798245"
Jul 09 18:05:45.457599 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.457559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:45.457599 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.457601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:45.458060 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.457629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:45.458060 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.457677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:45.458060 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:45.457798    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:45.458060 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:05:45.457872    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls podName:53d9f28c-f462-41ba-a0e8-b376d1d7a962 nodeName:}" failed. No retries permitted until 2026-07-09 18:06:01.45785229 +0000 UTC m=+81.176115055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ggcq4" (UID: "53d9f28c-f462-41ba-a0e8-b376d1d7a962") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:05:45.458191 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.458130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53d9f28c-f462-41ba-a0e8-b376d1d7a962-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:05:45.460805 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.460787    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:45.461746 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.461731    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:45.470007 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.469986    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e1d619-e463-49f5-b06c-f55c9cb8d1f2-metrics-certs\") pod \"network-metrics-daemon-jn9vx\" (UID: \"52e1d619-e463-49f5-b06c-f55c9cb8d1f2\") " pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:45.470109 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.470048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe991f9-27b4-4d56-a779-014879464f58-original-pull-secret\") pod \"global-pull-secret-syncer-lznbs\" (UID: \"0fe991f9-27b4-4d56-a779-014879464f58\") " pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:45.474952 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.474936    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lznbs"
Jul 09 18:05:45.491642 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.491623    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhwtc\""
Jul 09 18:05:45.499728 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.499705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jn9vx"
Jul 09 18:05:45.558642 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.558291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:45.562416 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.561683    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:45.572827 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.572771    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:45.582743 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.582717    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfh5n\" (UniqueName: \"kubernetes.io/projected/ec04d48c-c71a-4abd-85be-a98ff75e17a1-kube-api-access-vfh5n\") pod \"network-check-target-h2cxx\" (UID: \"ec04d48c-c71a-4abd-85be-a98ff75e17a1\") " pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:45.610356 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.610321    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lznbs"]
Jul 09 18:05:45.613413 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:05:45.613389    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe991f9_27b4_4d56_a779_014879464f58.slice/crio-fe7692639d166acf37aa5e5f2b4e37b61d276a480f88ba362591228e1d6fd615 WatchSource:0}: Error finding container fe7692639d166acf37aa5e5f2b4e37b61d276a480f88ba362591228e1d6fd615: Status 404 returned error can't find the container with id fe7692639d166acf37aa5e5f2b4e37b61d276a480f88ba362591228e1d6fd615
Jul 09 18:05:45.632443 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.632417    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jn9vx"]
Jul 09 18:05:45.635934 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:05:45.635894    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e1d619_e463_49f5_b06c_f55c9cb8d1f2.slice/crio-044fc6c045dc29c51277c99ac7dd60f71941d8bf669e63cd78a99f85fa2db0e5 WatchSource:0}: Error finding container 044fc6c045dc29c51277c99ac7dd60f71941d8bf669e63cd78a99f85fa2db0e5: Status 404 returned error can't find the container with id 044fc6c045dc29c51277c99ac7dd60f71941d8bf669e63cd78a99f85fa2db0e5
Jul 09 18:05:45.784806 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.784724    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcghf\""
Jul 09 18:05:45.792119 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.792100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:45.922283 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:45.922255    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h2cxx"]
Jul 09 18:05:45.925507 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:05:45.925475    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec04d48c_c71a_4abd_85be_a98ff75e17a1.slice/crio-3039ede4084fad7e590b2f92ced2741c88742e6b0f95c110663e92caeacebb42 WatchSource:0}: Error finding container 3039ede4084fad7e590b2f92ced2741c88742e6b0f95c110663e92caeacebb42: Status 404 returned error can't find the container with id 3039ede4084fad7e590b2f92ced2741c88742e6b0f95c110663e92caeacebb42
Jul 09 18:05:46.164233 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:46.164141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lznbs" event={"ID":"0fe991f9-27b4-4d56-a779-014879464f58","Type":"ContainerStarted","Data":"fe7692639d166acf37aa5e5f2b4e37b61d276a480f88ba362591228e1d6fd615"}
Jul 09 18:05:46.165630 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:46.165555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h2cxx" event={"ID":"ec04d48c-c71a-4abd-85be-a98ff75e17a1","Type":"ContainerStarted","Data":"3039ede4084fad7e590b2f92ced2741c88742e6b0f95c110663e92caeacebb42"}
Jul 09 18:05:46.166843 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:46.166816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jn9vx" event={"ID":"52e1d619-e463-49f5-b06c-f55c9cb8d1f2","Type":"ContainerStarted","Data":"044fc6c045dc29c51277c99ac7dd60f71941d8bf669e63cd78a99f85fa2db0e5"}
Jul 09 18:05:47.171411 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:47.171367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jn9vx" event={"ID":"52e1d619-e463-49f5-b06c-f55c9cb8d1f2","Type":"ContainerStarted","Data":"34b50985b7ad3c2876d29ce2b3092348b9514b87bd0ebfbc3d3dddbf9614f5ce"}
Jul 09 18:05:47.171791 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:47.171424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jn9vx" event={"ID":"52e1d619-e463-49f5-b06c-f55c9cb8d1f2","Type":"ContainerStarted","Data":"a5fc45e1fe75e294ed85fd40dbae986caa79868f761c316d9e115e1610449fe1"}
Jul 09 18:05:47.200887 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:47.200833    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jn9vx" podStartSLOduration=66.323829323 podStartE2EDuration="1m7.200811446s" totalImagesPullingTime="876.982123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:47.199674374 +0000 UTC m=+66.917937140" watchObservedRunningTime="2026-07-09 18:05:47.200811446 +0000 UTC m=+66.919074211"
Jul 09 18:05:48.080955 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:48.080900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:48.083629 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:48.083603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3656bece-edae-4e6d-b42a-7adca0774519-metrics-tls\") pod \"dns-default-vvfdf\" (UID: \"3656bece-edae-4e6d-b42a-7adca0774519\") " pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:48.295836 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:48.295804    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nq7m5\""
Jul 09 18:05:48.304209 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:48.304168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:50.957562 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:50.957534    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vvfdf"]
Jul 09 18:05:50.960579 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:05:50.960548    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3656bece_edae_4e6d_b42a_7adca0774519.slice/crio-8f89142a0b3176f6cafa0e8629f683690b4d1ab2f97ea87433affb86e75bed4d WatchSource:0}: Error finding container 8f89142a0b3176f6cafa0e8629f683690b4d1ab2f97ea87433affb86e75bed4d: Status 404 returned error can't find the container with id 8f89142a0b3176f6cafa0e8629f683690b4d1ab2f97ea87433affb86e75bed4d
Jul 09 18:05:51.185494 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:51.185388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lznbs" event={"ID":"0fe991f9-27b4-4d56-a779-014879464f58","Type":"ContainerStarted","Data":"79be0d840e7a44d1351e820c8dd0ece988bf167a0a64d9cd345b87656ae95ecc"}
Jul 09 18:05:51.186837 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:51.186810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h2cxx" event={"ID":"ec04d48c-c71a-4abd-85be-a98ff75e17a1","Type":"ContainerStarted","Data":"14dc438ed0252113fd90d7262a5ac26fc5cd12b330ada501765906f799b782f1"}
Jul 09 18:05:51.186972 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:51.186868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:05:51.187862 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:51.187835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvfdf" event={"ID":"3656bece-edae-4e6d-b42a-7adca0774519","Type":"ContainerStarted","Data":"8f89142a0b3176f6cafa0e8629f683690b4d1ab2f97ea87433affb86e75bed4d"}
Jul 09 18:05:51.203157 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:51.203114    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lznbs" podStartSLOduration=64.996187518 podStartE2EDuration="1m10.203100498s" totalImagesPullingTime="5.20691298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:51.202448123 +0000 UTC m=+70.920710886" watchObservedRunningTime="2026-07-09 18:05:51.203100498 +0000 UTC m=+70.921363262"
Jul 09 18:05:51.219618 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:51.219555    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h2cxx" podStartSLOduration=66.316058068 podStartE2EDuration="1m11.219543382s" totalImagesPullingTime="4.903485314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:51.218463997 +0000 UTC m=+70.936726760" watchObservedRunningTime="2026-07-09 18:05:51.219543382 +0000 UTC m=+70.937806145"
Jul 09 18:05:53.196697 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:53.196665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvfdf" event={"ID":"3656bece-edae-4e6d-b42a-7adca0774519","Type":"ContainerStarted","Data":"f0a5b0586e3e1db2ff96a66fd4ca034a441a64964216528feb3ea2a20427d6b5"}
Jul 09 18:05:53.197125 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:53.196715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvfdf" event={"ID":"3656bece-edae-4e6d-b42a-7adca0774519","Type":"ContainerStarted","Data":"5c60c243f1dbb17b5978f1ab8eb1c9c9979975606575c4ad51f2ff558a257a32"}
Jul 09 18:05:53.197125 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:53.196800    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:05:53.220188 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:53.220146    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vvfdf" podStartSLOduration=35.736106721 podStartE2EDuration="37.220129119s" totalImagesPullingTime="1.484022398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:53.218470287 +0000 UTC m=+72.936733085" watchObservedRunningTime="2026-07-09 18:05:53.220129119 +0000 UTC m=+72.938391882"
Jul 09 18:05:56.055612 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.055582    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-92h5k"]
Jul 09 18:05:56.057460 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.057442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-92h5k"
Jul 09 18:05:56.062210 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.062127    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-srptr\""
Jul 09 18:05:56.064539 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.064522    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 18:05:56.071059 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.071042    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 18:05:56.076991 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.076968    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-92h5k"]
Jul 09 18:05:56.132170 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.132140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvbc\" (UniqueName: \"kubernetes.io/projected/edb7714a-acfa-4ee9-b6ad-0579ce172655-kube-api-access-hxvbc\") pod \"downloads-68c9cbb497-92h5k\" (UID: \"edb7714a-acfa-4ee9-b6ad-0579ce172655\") " pod="openshift-console/downloads-68c9cbb497-92h5k"
Jul 09 18:05:56.140589 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.140559    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55dc77cc58-ckbs6"]
Jul 09 18:05:56.143076 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.143050    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.146769 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.146738    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:05:56.146769 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.146759    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:05:56.146968 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.146925    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4mr58\""
Jul 09 18:05:56.147055 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.147038    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:05:56.152259 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.152242    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:05:56.161653 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.161630    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55dc77cc58-ckbs6"]
Jul 09 18:05:56.233118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.233076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-ca-trust-extracted\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.233118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.233122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgsvj\" (UniqueName: \"kubernetes.io/projected/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-kube-api-access-tgsvj\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.233336 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.233177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-image-registry-private-configuration\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.233336 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.233261    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-registry-tls\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.233336 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.233301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvbc\" (UniqueName: \"kubernetes.io/projected/edb7714a-acfa-4ee9-b6ad-0579ce172655-kube-api-access-hxvbc\") pod \"downloads-68c9cbb497-92h5k\" (UID: \"edb7714a-acfa-4ee9-b6ad-0579ce172655\") " pod="openshift-console/downloads-68c9cbb497-92h5k"
Jul 09 18:05:56.233336 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.233323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-registry-certificates\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.233458 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.233339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-bound-sa-token\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.233458 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.233362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-installation-pull-secrets\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.233458 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.233422    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-trusted-ca\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.242317 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.242296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvbc\" (UniqueName: \"kubernetes.io/projected/edb7714a-acfa-4ee9-b6ad-0579ce172655-kube-api-access-hxvbc\") pod \"downloads-68c9cbb497-92h5k\" (UID: \"edb7714a-acfa-4ee9-b6ad-0579ce172655\") " pod="openshift-console/downloads-68c9cbb497-92h5k"
Jul 09 18:05:56.334656 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.334571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-installation-pull-secrets\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.334656 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.334622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-trusted-ca\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.334656 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.334654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-ca-trust-extracted\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.334873 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.334678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsvj\" (UniqueName: \"kubernetes.io/projected/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-kube-api-access-tgsvj\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.334873 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.334719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-image-registry-private-configuration\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.334873 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.334747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-registry-tls\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.334873 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.334792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-registry-certificates\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.334873 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.334817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-bound-sa-token\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.335208 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.335182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-ca-trust-extracted\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.335772 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.335745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-registry-certificates\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.335900 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.335884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-trusted-ca\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.337252 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.337221    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-installation-pull-secrets\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.337528 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.337508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-registry-tls\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.337607 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.337559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-image-registry-private-configuration\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.343625 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.343597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsvj\" (UniqueName: \"kubernetes.io/projected/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-kube-api-access-tgsvj\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.344243 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.344227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e9d83f7-2520-4ba1-ae4b-e924fe978f9d-bound-sa-token\") pod \"image-registry-55dc77cc58-ckbs6\" (UID: \"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d\") " pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.365409 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.365386    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-92h5k"
Jul 09 18:05:56.452114 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.452086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:56.494239 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.493424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-92h5k"]
Jul 09 18:05:56.495857 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:05:56.495833    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb7714a_acfa_4ee9_b6ad_0579ce172655.slice/crio-cae773e5966a35bf567d27b7794361e18c5df759729450d6f69acdd8584edf66 WatchSource:0}: Error finding container cae773e5966a35bf567d27b7794361e18c5df759729450d6f69acdd8584edf66: Status 404 returned error can't find the container with id cae773e5966a35bf567d27b7794361e18c5df759729450d6f69acdd8584edf66
Jul 09 18:05:56.586329 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:56.586251    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55dc77cc58-ckbs6"]
Jul 09 18:05:56.589087 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:05:56.589062    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9d83f7_2520_4ba1_ae4b_e924fe978f9d.slice/crio-9cd8591f4851b8675a13765db3ec66bc83a57487c6dbe2b59375ff0075f39aca WatchSource:0}: Error finding container 9cd8591f4851b8675a13765db3ec66bc83a57487c6dbe2b59375ff0075f39aca: Status 404 returned error can't find the container with id 9cd8591f4851b8675a13765db3ec66bc83a57487c6dbe2b59375ff0075f39aca
Jul 09 18:05:57.210751 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:57.210705    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6" event={"ID":"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d","Type":"ContainerStarted","Data":"c4c8d2839f797a3f150c335e341882c780025074498d31aa0e7a3b49bb0a7257"}
Jul 09 18:05:57.211239 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:57.210778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6" event={"ID":"2e9d83f7-2520-4ba1-ae4b-e924fe978f9d","Type":"ContainerStarted","Data":"9cd8591f4851b8675a13765db3ec66bc83a57487c6dbe2b59375ff0075f39aca"}
Jul 09 18:05:57.211239 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:57.210807    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:05:57.212139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:57.212114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-92h5k" event={"ID":"edb7714a-acfa-4ee9-b6ad-0579ce172655","Type":"ContainerStarted","Data":"cae773e5966a35bf567d27b7794361e18c5df759729450d6f69acdd8584edf66"}
Jul 09 18:05:57.235601 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:05:57.235534    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6" podStartSLOduration=1.235511341 podStartE2EDuration="1.235511341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:57.234365575 +0000 UTC m=+76.952628368" watchObservedRunningTime="2026-07-09 18:05:57.235511341 +0000 UTC m=+76.953774106"
Jul 09 18:06:01.476793 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:01.476651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:06:01.479436 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:01.479400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53d9f28c-f462-41ba-a0e8-b376d1d7a962-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggcq4\" (UID: \"53d9f28c-f462-41ba-a0e8-b376d1d7a962\") " pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:06:01.705529 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:01.705497    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qh478\""
Jul 09 18:06:01.712826 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:01.712804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ggcq4"
Jul 09 18:06:01.869603 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:01.869400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ggcq4"]
Jul 09 18:06:01.871935 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:06:01.871890    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d9f28c_f462_41ba_a0e8_b376d1d7a962.slice/crio-c32b152f1622248307b253a4b1d86f7d9099fc78afba23ffa968a843aa07750b WatchSource:0}: Error finding container c32b152f1622248307b253a4b1d86f7d9099fc78afba23ffa968a843aa07750b: Status 404 returned error can't find the container with id c32b152f1622248307b253a4b1d86f7d9099fc78afba23ffa968a843aa07750b
Jul 09 18:06:02.232465 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:02.232425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggcq4" event={"ID":"53d9f28c-f462-41ba-a0e8-b376d1d7a962","Type":"ContainerStarted","Data":"c32b152f1622248307b253a4b1d86f7d9099fc78afba23ffa968a843aa07750b"}
Jul 09 18:06:03.202234 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.202133    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vvfdf"
Jul 09 18:06:03.259242 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.258446    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b8889b578-k46f4"]
Jul 09 18:06:03.261337 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.260812    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.263620 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.263590    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 18:06:03.263743 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.263669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 18:06:03.264108 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.263816    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 18:06:03.264108 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.264044    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 18:06:03.264286 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.264134    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 18:06:03.264286 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.264183    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kdcn2\""
Jul 09 18:06:03.269855 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.269833    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 18:06:03.277095 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.277074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b8889b578-k46f4"]
Jul 09 18:06:03.291692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.290623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-serving-cert\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.291692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.290663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-oauth-config\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.291692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.290728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-service-ca\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.291692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.290772    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnf7f\" (UniqueName: \"kubernetes.io/projected/f81e44ea-cdbd-4a6e-b433-c454b71118d9-kube-api-access-pnf7f\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.291692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.290804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-trusted-ca-bundle\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.291692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.290846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-oauth-serving-cert\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.291692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.290869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-config\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.392049 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.392019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-service-ca\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.392244 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.392074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnf7f\" (UniqueName: \"kubernetes.io/projected/f81e44ea-cdbd-4a6e-b433-c454b71118d9-kube-api-access-pnf7f\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.392244 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.392108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-trusted-ca-bundle\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.392244 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.392136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-oauth-serving-cert\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.392244 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.392161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-config\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.392244 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.392193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-serving-cert\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.392244 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.392217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-oauth-config\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.392898 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.392870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-service-ca\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.393475 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.393428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-oauth-serving-cert\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.393863 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.393821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-config\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.394168 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.394118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-trusted-ca-bundle\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.395628 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.395587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-serving-cert\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.395727 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.395650    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-oauth-config\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.401256 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.401231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnf7f\" (UniqueName: \"kubernetes.io/projected/f81e44ea-cdbd-4a6e-b433-c454b71118d9-kube-api-access-pnf7f\") pod \"console-7b8889b578-k46f4\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") " pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.574565 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.574444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:03.728176 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:03.728140    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b8889b578-k46f4"]
Jul 09 18:06:05.091101 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:06:05.091056    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81e44ea_cdbd_4a6e_b433_c454b71118d9.slice/crio-6576c6802641baa6c0855d08c3d7fe6a2dd600d69929e4c2fd8a68980607a97c WatchSource:0}: Error finding container 6576c6802641baa6c0855d08c3d7fe6a2dd600d69929e4c2fd8a68980607a97c: Status 404 returned error can't find the container with id 6576c6802641baa6c0855d08c3d7fe6a2dd600d69929e4c2fd8a68980607a97c
Jul 09 18:06:05.246939 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:05.246878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8889b578-k46f4" event={"ID":"f81e44ea-cdbd-4a6e-b433-c454b71118d9","Type":"ContainerStarted","Data":"6576c6802641baa6c0855d08c3d7fe6a2dd600d69929e4c2fd8a68980607a97c"}
Jul 09 18:06:15.284484 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:15.284394    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggcq4" event={"ID":"53d9f28c-f462-41ba-a0e8-b376d1d7a962","Type":"ContainerStarted","Data":"5af3bbb12660c32eb0ded34ed2441baddc92400acf191deb2b30f8128bd49130"}
Jul 09 18:06:15.286389 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:15.286355    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8889b578-k46f4" event={"ID":"f81e44ea-cdbd-4a6e-b433-c454b71118d9","Type":"ContainerStarted","Data":"fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8"}
Jul 09 18:06:15.288295 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:15.288265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-92h5k" event={"ID":"edb7714a-acfa-4ee9-b6ad-0579ce172655","Type":"ContainerStarted","Data":"6b7017b51d8f9d48b0d903d079a297f85fb4f4defdcc94d5a8419c89378d4d22"}
Jul 09 18:06:15.288490 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:15.288474    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-92h5k"
Jul 09 18:06:15.311431 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:15.311361    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b8889b578-k46f4" podStartSLOduration=2.506138739 podStartE2EDuration="12.311340551s" totalImagesPullingTime="9.805201812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:15.310441526 +0000 UTC m=+95.028704291" watchObservedRunningTime="2026-07-09 18:06:15.311340551 +0000 UTC m=+95.029603316"
Jul 09 18:06:15.312311 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:15.312286    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-92h5k"
Jul 09 18:06:15.332094 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:15.331168    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-92h5k" podStartSLOduration=1.096113597 podStartE2EDuration="19.331147182s" totalImagesPullingTime="18.235033585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:15.329634726 +0000 UTC m=+95.047897503" watchObservedRunningTime="2026-07-09 18:06:15.331147182 +0000 UTC m=+95.049410040"
Jul 09 18:06:16.295206 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:16.295166    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggcq4" event={"ID":"53d9f28c-f462-41ba-a0e8-b376d1d7a962","Type":"ContainerStarted","Data":"ce3b0df036fcd7aae7bed496119dd6c78ea6d2ddabae3749c721e5b8eca418b5"}
Jul 09 18:06:18.223715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:18.223678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55dc77cc58-ckbs6"
Jul 09 18:06:20.315431 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:20.315389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggcq4" event={"ID":"53d9f28c-f462-41ba-a0e8-b376d1d7a962","Type":"ContainerStarted","Data":"6bded430a43397be11a845f547a5910525f53aedb33217ed5f3b2fcc4aa1dc1a"}
Jul 09 18:06:20.348456 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:20.348403    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ggcq4" podStartSLOduration=34.003330197 podStartE2EDuration="51.348386625s" totalImagesPullingTime="17.345056428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:20.347772802 +0000 UTC m=+100.066035569" watchObservedRunningTime="2026-07-09 18:06:20.348386625 +0000 UTC m=+100.066649389"
Jul 09 18:06:22.194666 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:22.194632    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h2cxx"
Jul 09 18:06:23.575501 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:23.575454    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:23.575979 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:23.575567    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:23.581241 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:23.581218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:24.335720 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:24.335694    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:06:28.852712 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.852676    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"]
Jul 09 18:06:28.894482 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.894447    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:28.897347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.897321    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 09 18:06:28.898619 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.898560    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xxtcm\""
Jul 09 18:06:28.898619 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.898580    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:06:28.898619 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.898591    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 09 18:06:28.898835 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.898569    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:06:28.898835 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.898592    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:06:28.898835 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.898584    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 09 18:06:28.898835 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.898818    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v2zkd"]
Jul 09 18:06:28.924394 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.924365    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"]
Jul 09 18:06:28.924525 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.924481    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:28.927171 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.927149    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:06:28.927171 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.927162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:06:28.927328 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.927153    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hpgbg\""
Jul 09 18:06:28.927328 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:28.927150    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:06:29.003755 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.003727    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.003919 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.003761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.003919 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.003784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.003919 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.003857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-wtmp\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.003919 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.003899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.004058 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.003937    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnrr\" (UniqueName: \"kubernetes.io/projected/b5b9872b-eb91-444f-8522-8c0007eb185e-kube-api-access-kdnrr\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.004058 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.003960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-tls\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.004058 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.003981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b9872b-eb91-444f-8522-8c0007eb185e-root\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.004058 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.004024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-textfile\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.004058 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.004057    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dg7\" (UniqueName: \"kubernetes.io/projected/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-api-access-m5dg7\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.004207 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.004073    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.004207 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.004091    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9872b-eb91-444f-8522-8c0007eb185e-metrics-client-ca\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.004207 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.004106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.004207 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.004132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b9872b-eb91-444f-8522-8c0007eb185e-sys\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.004207 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.004146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105137 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-wtmp\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105137 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.105137 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnrr\" (UniqueName: \"kubernetes.io/projected/b5b9872b-eb91-444f-8522-8c0007eb185e-kube-api-access-kdnrr\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105137 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-tls\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b9872b-eb91-444f-8522-8c0007eb185e-root\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-wtmp\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-textfile\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b9872b-eb91-444f-8522-8c0007eb185e-root\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dg7\" (UniqueName: \"kubernetes.io/projected/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-api-access-m5dg7\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.105705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.105705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9872b-eb91-444f-8522-8c0007eb185e-metrics-client-ca\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-textfile\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b9872b-eb91-444f-8522-8c0007eb185e-sys\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.105705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.106043 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b9872b-eb91-444f-8522-8c0007eb185e-sys\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.106043 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.106043 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.106043 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.106043 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.106043 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.105960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.106043 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:06:29.105986    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 09 18:06:29.106337 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:06:29.106057    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-tls podName:ec32bc36-b9ed-479c-a795-0f6c735fd0ab nodeName:}" failed. No retries permitted until 2026-07-09 18:06:29.606037522 +0000 UTC m=+109.324300278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-g7hw9" (UID: "ec32bc36-b9ed-479c-a795-0f6c735fd0ab") : secret "kube-state-metrics-tls" not found
Jul 09 18:06:29.106337 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.106110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9872b-eb91-444f-8522-8c0007eb185e-metrics-client-ca\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.106337 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.106205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.106551 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.106532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.107752 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.107732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-tls\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.108326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.108310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b9872b-eb91-444f-8522-8c0007eb185e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.108487 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.108467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.114236 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.114214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dg7\" (UniqueName: \"kubernetes.io/projected/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-api-access-m5dg7\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.114326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.114269    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnrr\" (UniqueName: \"kubernetes.io/projected/b5b9872b-eb91-444f-8522-8c0007eb185e-kube-api-access-kdnrr\") pod \"node-exporter-v2zkd\" (UID: \"b5b9872b-eb91-444f-8522-8c0007eb185e\") " pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.232681 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.232640    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2zkd"
Jul 09 18:06:29.274923 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:06:29.274877    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b9872b_eb91_444f_8522_8c0007eb185e.slice/crio-0b4eff5eeb2241e1b79398f8161c74323fe38e26db91f4f8f8ade2cb32285c71 WatchSource:0}: Error finding container 0b4eff5eeb2241e1b79398f8161c74323fe38e26db91f4f8f8ade2cb32285c71: Status 404 returned error can't find the container with id 0b4eff5eeb2241e1b79398f8161c74323fe38e26db91f4f8f8ade2cb32285c71
Jul 09 18:06:29.348374 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.348339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2zkd" event={"ID":"b5b9872b-eb91-444f-8522-8c0007eb185e","Type":"ContainerStarted","Data":"0b4eff5eeb2241e1b79398f8161c74323fe38e26db91f4f8f8ade2cb32285c71"}
Jul 09 18:06:29.610454 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.610424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.612770 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.612749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec32bc36-b9ed-479c-a795-0f6c735fd0ab-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-g7hw9\" (UID: \"ec32bc36-b9ed-479c-a795-0f6c735fd0ab\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.803693 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.803661    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"
Jul 09 18:06:29.962894 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:29.962842    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-g7hw9"]
Jul 09 18:06:29.966273 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:06:29.966230    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec32bc36_b9ed_479c_a795_0f6c735fd0ab.slice/crio-c782c182dbfcb31f2f6c2036ba577f48e0eb78d94f5b8dfdb6ae57f31adcd031 WatchSource:0}: Error finding container c782c182dbfcb31f2f6c2036ba577f48e0eb78d94f5b8dfdb6ae57f31adcd031: Status 404 returned error can't find the container with id c782c182dbfcb31f2f6c2036ba577f48e0eb78d94f5b8dfdb6ae57f31adcd031
Jul 09 18:06:30.354056 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:30.354018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9" event={"ID":"ec32bc36-b9ed-479c-a795-0f6c735fd0ab","Type":"ContainerStarted","Data":"c782c182dbfcb31f2f6c2036ba577f48e0eb78d94f5b8dfdb6ae57f31adcd031"}
Jul 09 18:06:31.358723 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:31.358628    2565 generic.go:356] "Generic (PLEG): container finished" podID="b5b9872b-eb91-444f-8522-8c0007eb185e" containerID="3a98752b3e6560ffb52746529e274c41c57f452c9e692a19498ea936f6c0e318" exitCode=0
Jul 09 18:06:31.359139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:31.358711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2zkd" event={"ID":"b5b9872b-eb91-444f-8522-8c0007eb185e","Type":"ContainerDied","Data":"3a98752b3e6560ffb52746529e274c41c57f452c9e692a19498ea936f6c0e318"}
Jul 09 18:06:32.364925 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:32.364869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2zkd" event={"ID":"b5b9872b-eb91-444f-8522-8c0007eb185e","Type":"ContainerStarted","Data":"f427cdeac893ddc9c5e1fef7d9e75cc96289bb53c3431dc1d8aa8bd03b768a51"}
Jul 09 18:06:32.365388 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:32.364968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2zkd" event={"ID":"b5b9872b-eb91-444f-8522-8c0007eb185e","Type":"ContainerStarted","Data":"14393f126aad35e835c0016785cf6a2dd76bb56b7ca067acb4a865caf1bc7292"}
Jul 09 18:06:32.391423 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:32.391373    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v2zkd" podStartSLOduration=2.615043494 podStartE2EDuration="4.391357736s" totalImagesPullingTime="1.683312229s" totalInitContainerRuntime="93.002013ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:32.390168025 +0000 UTC m=+112.108430790" watchObservedRunningTime="2026-07-09 18:06:32.391357736 +0000 UTC m=+112.109620519"
Jul 09 18:06:33.544391 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.544349    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"]
Jul 09 18:06:33.563259 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.563233    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"]
Jul 09 18:06:33.563425 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.563321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:33.566253 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.566230    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 09 18:06:33.566253 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.566242    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bq7nl\""
Jul 09 18:06:33.742047 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.742012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhps\" (UniqueName: \"kubernetes.io/projected/f3ab618e-407f-49ca-a5eb-68262997db9e-kube-api-access-bqhps\") pod \"monitoring-plugin-5f68df7f58-t78pw\" (UID: \"f3ab618e-407f-49ca-a5eb-68262997db9e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:33.742204 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.742082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3ab618e-407f-49ca-a5eb-68262997db9e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t78pw\" (UID: \"f3ab618e-407f-49ca-a5eb-68262997db9e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:33.843467 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.843395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhps\" (UniqueName: \"kubernetes.io/projected/f3ab618e-407f-49ca-a5eb-68262997db9e-kube-api-access-bqhps\") pod \"monitoring-plugin-5f68df7f58-t78pw\" (UID: \"f3ab618e-407f-49ca-a5eb-68262997db9e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:33.843467 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.843461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3ab618e-407f-49ca-a5eb-68262997db9e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t78pw\" (UID: \"f3ab618e-407f-49ca-a5eb-68262997db9e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:33.845748 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.845726    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3ab618e-407f-49ca-a5eb-68262997db9e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t78pw\" (UID: \"f3ab618e-407f-49ca-a5eb-68262997db9e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:33.852244 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.852220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhps\" (UniqueName: \"kubernetes.io/projected/f3ab618e-407f-49ca-a5eb-68262997db9e-kube-api-access-bqhps\") pod \"monitoring-plugin-5f68df7f58-t78pw\" (UID: \"f3ab618e-407f-49ca-a5eb-68262997db9e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:33.871672 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.871654    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:33.977358 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.977329    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"]
Jul 09 18:06:33.993992 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.993964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:33.994320 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.994286    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"]
Jul 09 18:06:33.997163 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.997020    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 09 18:06:33.997163 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.997041    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 09 18:06:33.997163 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.997085    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 09 18:06:33.997375 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.997204    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8kmwd\""
Jul 09 18:06:33.997375 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.997204    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 09 18:06:33.997375 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:33.997370    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 09 18:06:34.002653 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.002633    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 09 18:06:34.121705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.121629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"]
Jul 09 18:06:34.124890 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:06:34.124861    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ab618e_407f_49ca_a5eb_68262997db9e.slice/crio-afed4f7daa4f4df89e3a9728ae46669dafdce7f0149f3c4c6f2dde0ad9787bb7 WatchSource:0}: Error finding container afed4f7daa4f4df89e3a9728ae46669dafdce7f0149f3c4c6f2dde0ad9787bb7: Status 404 returned error can't find the container with id afed4f7daa4f4df89e3a9728ae46669dafdce7f0149f3c4c6f2dde0ad9787bb7
Jul 09 18:06:34.146588 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.146561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-telemeter-client-tls\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.146732 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.146614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.146775 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.146731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e24db2ac-e7dd-453b-950b-6ebb00f3952e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.146839 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.146789    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e24db2ac-e7dd-453b-950b-6ebb00f3952e-serving-certs-ca-bundle\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.146839 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.146822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-federate-client-tls\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.146957 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.146853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e24db2ac-e7dd-453b-950b-6ebb00f3952e-metrics-client-ca\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.146957 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.146876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwhp\" (UniqueName: \"kubernetes.io/projected/e24db2ac-e7dd-453b-950b-6ebb00f3952e-kube-api-access-vnwhp\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.146957 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.146944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-secret-telemeter-client\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.248124 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.248088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.248302 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.248139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e24db2ac-e7dd-453b-950b-6ebb00f3952e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.248302 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.248170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e24db2ac-e7dd-453b-950b-6ebb00f3952e-serving-certs-ca-bundle\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.248302 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.248198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-federate-client-tls\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.248302 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.248228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e24db2ac-e7dd-453b-950b-6ebb00f3952e-metrics-client-ca\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.248302 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.248256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwhp\" (UniqueName: \"kubernetes.io/projected/e24db2ac-e7dd-453b-950b-6ebb00f3952e-kube-api-access-vnwhp\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.248560 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.248305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-secret-telemeter-client\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.248560 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.248338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-telemeter-client-tls\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.249083 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.249031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e24db2ac-e7dd-453b-950b-6ebb00f3952e-serving-certs-ca-bundle\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.249201 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.249113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e24db2ac-e7dd-453b-950b-6ebb00f3952e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.249201 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.249126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e24db2ac-e7dd-453b-950b-6ebb00f3952e-metrics-client-ca\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.251632 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.251417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.251732 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.251630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-telemeter-client-tls\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.251896 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.251875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-federate-client-tls\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.252014 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.251991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e24db2ac-e7dd-453b-950b-6ebb00f3952e-secret-telemeter-client\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.260030 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.260005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwhp\" (UniqueName: \"kubernetes.io/projected/e24db2ac-e7dd-453b-950b-6ebb00f3952e-kube-api-access-vnwhp\") pod \"telemeter-client-5d95b9d954-vjbdq\" (UID: \"e24db2ac-e7dd-453b-950b-6ebb00f3952e\") " pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.314888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.314861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"
Jul 09 18:06:34.374147 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.374063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9" event={"ID":"ec32bc36-b9ed-479c-a795-0f6c735fd0ab","Type":"ContainerStarted","Data":"bcac370f760a1c26528ab002ae52e5da8a99a36f117a992c189064e82dfcd61b"}
Jul 09 18:06:34.374147 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.374137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9" event={"ID":"ec32bc36-b9ed-479c-a795-0f6c735fd0ab","Type":"ContainerStarted","Data":"37a89e6110c336768e130d365cabaf311624e9156b797a1cfb0699e8d269a707"}
Jul 09 18:06:34.375289 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.375262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw" event={"ID":"f3ab618e-407f-49ca-a5eb-68262997db9e","Type":"ContainerStarted","Data":"afed4f7daa4f4df89e3a9728ae46669dafdce7f0149f3c4c6f2dde0ad9787bb7"}
Jul 09 18:06:34.482894 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:34.482864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d95b9d954-vjbdq"]
Jul 09 18:06:34.486508 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:06:34.486481    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24db2ac_e7dd_453b_950b_6ebb00f3952e.slice/crio-4625aaba8f0e511b0afdf324b25d012132ba22a44a4151b9578ee3109c1d3648 WatchSource:0}: Error finding container 4625aaba8f0e511b0afdf324b25d012132ba22a44a4151b9578ee3109c1d3648: Status 404 returned error can't find the container with id 4625aaba8f0e511b0afdf324b25d012132ba22a44a4151b9578ee3109c1d3648
Jul 09 18:06:35.123883 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.123846    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:35.145559 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.145538    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:35.145725 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.145692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.149133 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.149108    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3p6107sntgr7u\""
Jul 09 18:06:35.149133 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.149108    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 18:06:35.149320 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.149130    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 18:06:35.149690 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.149669    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 18:06:35.149836 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.149809    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 18:06:35.150297 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.150279    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 18:06:35.150605 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.150586    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 18:06:35.150692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.150609    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 18:06:35.150692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.150668    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 18:06:35.151020 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.150864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sbkdk\""
Jul 09 18:06:35.152327 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.152310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 18:06:35.153558 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.152770    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 18:06:35.153558 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.152953    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 18:06:35.153558 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.153088    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 18:06:35.153558 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.153199    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 18:06:35.156272 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.156169    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 18:06:35.159095 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.159075    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 18:06:35.257505 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.257706 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257515    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.257706 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.257706 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aabeba0-a5f0-46e8-b6ce-790251d64736-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.257706 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.257706 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-web-config\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6bh\" (UniqueName: \"kubernetes.io/projected/9aabeba0-a5f0-46e8-b6ce-790251d64736-kube-api-access-wr6bh\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257859    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.257956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.258004    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258380 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.258048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258380 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.258074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258380 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.258103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aabeba0-a5f0-46e8-b6ce-790251d64736-config-out\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258380 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.258146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258380 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.258206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-config\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258380 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.258235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.258380 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.258257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359256 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-config\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359256 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359341    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aabeba0-a5f0-46e8-b6ce-790251d64736-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-web-config\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.359506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6bh\" (UniqueName: \"kubernetes.io/projected/9aabeba0-a5f0-46e8-b6ce-790251d64736-kube-api-access-wr6bh\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359736    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aabeba0-a5f0-46e8-b6ce-790251d64736-config-out\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360038 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.359786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360462 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.360152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360523 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.360489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.360579 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.360534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.362473 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.361473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.362473 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.362175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.362728 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.362704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.363291 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.363249    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.363291 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.363275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9aabeba0-a5f0-46e8-b6ce-790251d64736-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.364405 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.364208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.364405 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.364240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aabeba0-a5f0-46e8-b6ce-790251d64736-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.364671 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.364626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-config\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.364779 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.364676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.364948 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.364898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.365048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.364997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.365234 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.365192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-web-config\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.365379 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.365340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.365465 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.365391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aabeba0-a5f0-46e8-b6ce-790251d64736-config-out\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.365626 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.365602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.365724 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.365708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9aabeba0-a5f0-46e8-b6ce-790251d64736-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.370845 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.370824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6bh\" (UniqueName: \"kubernetes.io/projected/9aabeba0-a5f0-46e8-b6ce-790251d64736-kube-api-access-wr6bh\") pod \"prometheus-k8s-0\" (UID: \"9aabeba0-a5f0-46e8-b6ce-790251d64736\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.384559 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.384496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq" event={"ID":"e24db2ac-e7dd-453b-950b-6ebb00f3952e","Type":"ContainerStarted","Data":"4625aaba8f0e511b0afdf324b25d012132ba22a44a4151b9578ee3109c1d3648"}
Jul 09 18:06:35.386548 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.386525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9" event={"ID":"ec32bc36-b9ed-479c-a795-0f6c735fd0ab","Type":"ContainerStarted","Data":"46d1b7d03dd2a85a1cc0955de57d12e8bd0751f255d36491efcb2ed4b7827b64"}
Jul 09 18:06:35.405797 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.405733    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-g7hw9" podStartSLOduration=3.378761152 podStartE2EDuration="7.4057124s" totalImagesPullingTime="4.026951248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:35.405274258 +0000 UTC m=+115.123537036" watchObservedRunningTime="2026-07-09 18:06:35.4057124 +0000 UTC m=+115.123975165"
Jul 09 18:06:35.460148 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.460100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:35.650189 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:35.650111    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:36.038982 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:06:36.038946    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aabeba0_a5f0_46e8_b6ce_790251d64736.slice/crio-f147407be0da72d925b57d71d401a78972d6ed16b6f085f637b6df77a6f34665 WatchSource:0}: Error finding container f147407be0da72d925b57d71d401a78972d6ed16b6f085f637b6df77a6f34665: Status 404 returned error can't find the container with id f147407be0da72d925b57d71d401a78972d6ed16b6f085f637b6df77a6f34665
Jul 09 18:06:36.392255 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:36.392165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9aabeba0-a5f0-46e8-b6ce-790251d64736","Type":"ContainerStarted","Data":"f147407be0da72d925b57d71d401a78972d6ed16b6f085f637b6df77a6f34665"}
Jul 09 18:06:37.399548 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:37.399444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq" event={"ID":"e24db2ac-e7dd-453b-950b-6ebb00f3952e","Type":"ContainerStarted","Data":"118352cab8a4aaa53100512099a3b9f2ad5016e58337cc6ad64587f2ddc55789"}
Jul 09 18:06:37.401318 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:37.401288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw" event={"ID":"f3ab618e-407f-49ca-a5eb-68262997db9e","Type":"ContainerStarted","Data":"aefa8bd6f89bc9e274313dd21a084cd04d4e7eb09f3fe1b9f5b29603f93aa6d7"}
Jul 09 18:06:37.401564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:37.401514    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:37.407757 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:37.407737    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw"
Jul 09 18:06:37.420045 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:37.419992    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t78pw" podStartSLOduration=1.631690919 podStartE2EDuration="4.419976932s" totalImagesPullingTime="2.788286013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:37.418848848 +0000 UTC m=+117.137111613" watchObservedRunningTime="2026-07-09 18:06:37.419976932 +0000 UTC m=+117.138239696"
Jul 09 18:06:38.406661 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:38.406624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq" event={"ID":"e24db2ac-e7dd-453b-950b-6ebb00f3952e","Type":"ContainerStarted","Data":"b44367ee5d54196b68f14952edac888f0787109ce6aa472cbce43c62bc2ef60c"}
Jul 09 18:06:38.407139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:38.406726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq" event={"ID":"e24db2ac-e7dd-453b-950b-6ebb00f3952e","Type":"ContainerStarted","Data":"32ff50ff912bdc1267f399f66e85025bf9d901b2622468c48683e444c0e8fe93"}
Jul 09 18:06:38.408168 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:38.408147    2565 generic.go:356] "Generic (PLEG): container finished" podID="9aabeba0-a5f0-46e8-b6ce-790251d64736" containerID="1b70a5a3dcab485f912175a2be960fb12e9424cab9fb7440733e00e90638f4c5" exitCode=0
Jul 09 18:06:38.408243 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:38.408184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9aabeba0-a5f0-46e8-b6ce-790251d64736","Type":"ContainerDied","Data":"1b70a5a3dcab485f912175a2be960fb12e9424cab9fb7440733e00e90638f4c5"}
Jul 09 18:06:38.432575 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:38.432534    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d95b9d954-vjbdq" podStartSLOduration=2.01701373 podStartE2EDuration="5.432518416s" totalImagesPullingTime="3.415504686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:38.431276909 +0000 UTC m=+118.149539674" watchObservedRunningTime="2026-07-09 18:06:38.432518416 +0000 UTC m=+118.150781179"
Jul 09 18:06:43.431723 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:43.431690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9aabeba0-a5f0-46e8-b6ce-790251d64736","Type":"ContainerStarted","Data":"9044d3f6a1f2fa294d38b78345dbb308dcccdf012950af794c2f66f560241938"}
Jul 09 18:06:43.432235 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:43.431764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9aabeba0-a5f0-46e8-b6ce-790251d64736","Type":"ContainerStarted","Data":"ebe0ee69cc2f1ddc86aa222ac6c20076c8aa22c02680bc4e0815f7c030b75d3d"}
Jul 09 18:06:45.441325 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:45.441295    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9aabeba0-a5f0-46e8-b6ce-790251d64736","Type":"ContainerStarted","Data":"54f66eacadf2cff7dc350716fe8129fad001c3206b17dae3da35706338805c16"}
Jul 09 18:06:45.441666 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:45.441382    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9aabeba0-a5f0-46e8-b6ce-790251d64736","Type":"ContainerStarted","Data":"c28a15602b5037224c29611da1a609ae607586eca9ed6d7074720b1642365d1e"}
Jul 09 18:06:46.448164 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:46.448130    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9aabeba0-a5f0-46e8-b6ce-790251d64736","Type":"ContainerStarted","Data":"617c79ba65ca5c827f2cc501141ada736d6141251d418ec0ebeceb92b110cd98"}
Jul 09 18:06:46.448530 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:46.448194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9aabeba0-a5f0-46e8-b6ce-790251d64736","Type":"ContainerStarted","Data":"af161f2c493fe4ba3a67cd9245a865b38753507a054756a618811b1a01c29bd0"}
Jul 09 18:06:46.482744 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:46.482689    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.005455366 podStartE2EDuration="11.482670453s" totalImagesPullingTime="8.470043087s" totalInitContainerRuntime="7.172ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:46.479572557 +0000 UTC m=+126.197835323" watchObservedRunningTime="2026-07-09 18:06:46.482670453 +0000 UTC m=+126.200933216"
Jul 09 18:06:49.812501 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:49.812465    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b8889b578-k46f4"]
Jul 09 18:06:50.460332 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:06:50.460290    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:07:14.832457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:14.832406    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b8889b578-k46f4" podUID="f81e44ea-cdbd-4a6e-b433-c454b71118d9" containerName="console" containerID="cri-o://fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8" gracePeriod=15
Jul 09 18:07:15.067870 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.067844    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:07:15.196777 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.196748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-oauth-serving-cert\") pod \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") "
Jul 09 18:07:15.196975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.196790    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-serving-cert\") pod \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") "
Jul 09 18:07:15.196975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.196808    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-oauth-config\") pod \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") "
Jul 09 18:07:15.196975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.196877    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnf7f\" (UniqueName: \"kubernetes.io/projected/f81e44ea-cdbd-4a6e-b433-c454b71118d9-kube-api-access-pnf7f\") pod \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") "
Jul 09 18:07:15.196975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.196897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-config\") pod \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") "
Jul 09 18:07:15.196975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.196950    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-service-ca\") pod \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") "
Jul 09 18:07:15.197236 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.196981    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-trusted-ca-bundle\") pod \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\" (UID: \"f81e44ea-cdbd-4a6e-b433-c454b71118d9\") "
Jul 09 18:07:15.197417 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.197386    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f81e44ea-cdbd-4a6e-b433-c454b71118d9" (UID: "f81e44ea-cdbd-4a6e-b433-c454b71118d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:15.197417 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.197399    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-config" (OuterVolumeSpecName: "console-config") pod "f81e44ea-cdbd-4a6e-b433-c454b71118d9" (UID: "f81e44ea-cdbd-4a6e-b433-c454b71118d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:15.197521 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.197405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "f81e44ea-cdbd-4a6e-b433-c454b71118d9" (UID: "f81e44ea-cdbd-4a6e-b433-c454b71118d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:15.197521 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.197417    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f81e44ea-cdbd-4a6e-b433-c454b71118d9" (UID: "f81e44ea-cdbd-4a6e-b433-c454b71118d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:07:15.199211 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.199184    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f81e44ea-cdbd-4a6e-b433-c454b71118d9" (UID: "f81e44ea-cdbd-4a6e-b433-c454b71118d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:07:15.199306 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.199235    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f81e44ea-cdbd-4a6e-b433-c454b71118d9" (UID: "f81e44ea-cdbd-4a6e-b433-c454b71118d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:07:15.199306 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.199252    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81e44ea-cdbd-4a6e-b433-c454b71118d9-kube-api-access-pnf7f" (OuterVolumeSpecName: "kube-api-access-pnf7f") pod "f81e44ea-cdbd-4a6e-b433-c454b71118d9" (UID: "f81e44ea-cdbd-4a6e-b433-c454b71118d9"). InnerVolumeSpecName "kube-api-access-pnf7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:07:15.297801 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.297767    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.297801 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.297796    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-service-ca\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.297801 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.297808    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-trusted-ca-bundle\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.298064 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.297821    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f81e44ea-cdbd-4a6e-b433-c454b71118d9-oauth-serving-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.298064 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.297833    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-serving-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.298064 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.297844    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f81e44ea-cdbd-4a6e-b433-c454b71118d9-console-oauth-config\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.298064 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.297855    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnf7f\" (UniqueName: \"kubernetes.io/projected/f81e44ea-cdbd-4a6e-b433-c454b71118d9-kube-api-access-pnf7f\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:07:15.553457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.553367    2565 generic.go:356] "Generic (PLEG): container finished" podID="f81e44ea-cdbd-4a6e-b433-c454b71118d9" containerID="fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8" exitCode=0
Jul 09 18:07:15.553457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.553411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8889b578-k46f4"
Jul 09 18:07:15.553639 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.553442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8889b578-k46f4" event={"ID":"f81e44ea-cdbd-4a6e-b433-c454b71118d9","Type":"ContainerDied","Data":"fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8"}
Jul 09 18:07:15.553639 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.553529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8889b578-k46f4" event={"ID":"f81e44ea-cdbd-4a6e-b433-c454b71118d9","Type":"ContainerDied","Data":"6576c6802641baa6c0855d08c3d7fe6a2dd600d69929e4c2fd8a68980607a97c"}
Jul 09 18:07:15.553639 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.553561    2565 scope.go:117] "RemoveContainer" containerID="fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8"
Jul 09 18:07:15.562456 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.562438    2565 scope.go:117] "RemoveContainer" containerID="fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8"
Jul 09 18:07:15.562730 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:07:15.562709    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8\": container with ID starting with fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8 not found: ID does not exist" containerID="fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8"
Jul 09 18:07:15.562792 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.562740    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8"} err="failed to get container status \"fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8\": rpc error: code = NotFound desc = could not find container \"fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8\": container with ID starting with fe014ee22f609122d81beb639d6ffe17d24541f3b87151b9efe8dda01b4e53a8 not found: ID does not exist"
Jul 09 18:07:15.573304 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.573278    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b8889b578-k46f4"]
Jul 09 18:07:15.576965 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:15.576942    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b8889b578-k46f4"]
Jul 09 18:07:16.867300 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:16.867268    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f81e44ea-cdbd-4a6e-b433-c454b71118d9" path="/var/lib/kubelet/pods/f81e44ea-cdbd-4a6e-b433-c454b71118d9/volumes"
Jul 09 18:07:35.460584 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:35.460472    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:07:35.479798 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:35.479770    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:07:35.640341 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:07:35.640304    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:09:40.669954 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:09:40.669924    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:09:40.674276 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:09:40.674251    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:09:40.675755 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:09:40.675734    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:11:11.629464 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.629426    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7ttrf"]
Jul 09 18:11:11.629971 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.629711    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81e44ea-cdbd-4a6e-b433-c454b71118d9" containerName="console"
Jul 09 18:11:11.629971 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.629723    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81e44ea-cdbd-4a6e-b433-c454b71118d9" containerName="console"
Jul 09 18:11:11.629971 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.629779    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81e44ea-cdbd-4a6e-b433-c454b71118d9" containerName="console"
Jul 09 18:11:11.631560 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.631543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:11.635669 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.635648    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 09 18:11:11.635979 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.635962    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 09 18:11:11.641801 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.641777    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-q4d7x\""
Jul 09 18:11:11.652775 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.652748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7ttrf"]
Jul 09 18:11:11.698481 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.698445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x54tx\" (UniqueName: \"kubernetes.io/projected/27388a14-2da8-4e97-85b5-60b7c7e99170-kube-api-access-x54tx\") pod \"cert-manager-webhook-6486ffd997-7ttrf\" (UID: \"27388a14-2da8-4e97-85b5-60b7c7e99170\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:11.698656 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.698571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27388a14-2da8-4e97-85b5-60b7c7e99170-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7ttrf\" (UID: \"27388a14-2da8-4e97-85b5-60b7c7e99170\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:11.799303 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.799267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27388a14-2da8-4e97-85b5-60b7c7e99170-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7ttrf\" (UID: \"27388a14-2da8-4e97-85b5-60b7c7e99170\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:11.799303 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.799313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x54tx\" (UniqueName: \"kubernetes.io/projected/27388a14-2da8-4e97-85b5-60b7c7e99170-kube-api-access-x54tx\") pod \"cert-manager-webhook-6486ffd997-7ttrf\" (UID: \"27388a14-2da8-4e97-85b5-60b7c7e99170\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:11.823486 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.823446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27388a14-2da8-4e97-85b5-60b7c7e99170-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7ttrf\" (UID: \"27388a14-2da8-4e97-85b5-60b7c7e99170\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:11.823700 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.823684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x54tx\" (UniqueName: \"kubernetes.io/projected/27388a14-2da8-4e97-85b5-60b7c7e99170-kube-api-access-x54tx\") pod \"cert-manager-webhook-6486ffd997-7ttrf\" (UID: \"27388a14-2da8-4e97-85b5-60b7c7e99170\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:11.958529 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:11.958494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:12.103841 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:12.103811    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7ttrf"]
Jul 09 18:11:12.106931 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:11:12.106885    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27388a14_2da8_4e97_85b5_60b7c7e99170.slice/crio-ea50f96fb71f5a76522355ffc4963cfbb0e268d3289a9ed5240490783fb9cd69 WatchSource:0}: Error finding container ea50f96fb71f5a76522355ffc4963cfbb0e268d3289a9ed5240490783fb9cd69: Status 404 returned error can't find the container with id ea50f96fb71f5a76522355ffc4963cfbb0e268d3289a9ed5240490783fb9cd69
Jul 09 18:11:12.108857 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:12.108839    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:11:12.394264 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:12.394177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf" event={"ID":"27388a14-2da8-4e97-85b5-60b7c7e99170","Type":"ContainerStarted","Data":"ea50f96fb71f5a76522355ffc4963cfbb0e268d3289a9ed5240490783fb9cd69"}
Jul 09 18:11:13.432130 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.432084    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"]
Jul 09 18:11:13.434680 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.434658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"
Jul 09 18:11:13.437986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.437957    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-c7pjp\""
Jul 09 18:11:13.454439 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.454414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"]
Jul 09 18:11:13.514289 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.514251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7s5q\" (UniqueName: \"kubernetes.io/projected/c2d9d246-66bd-40db-b387-b3912c28537a-kube-api-access-g7s5q\") pod \"cert-manager-cainjector-559cdd8c87-kbvcg\" (UID: \"c2d9d246-66bd-40db-b387-b3912c28537a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"
Jul 09 18:11:13.514476 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.514311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d9d246-66bd-40db-b387-b3912c28537a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kbvcg\" (UID: \"c2d9d246-66bd-40db-b387-b3912c28537a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"
Jul 09 18:11:13.615171 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.615111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7s5q\" (UniqueName: \"kubernetes.io/projected/c2d9d246-66bd-40db-b387-b3912c28537a-kube-api-access-g7s5q\") pod \"cert-manager-cainjector-559cdd8c87-kbvcg\" (UID: \"c2d9d246-66bd-40db-b387-b3912c28537a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"
Jul 09 18:11:13.615171 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.615173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d9d246-66bd-40db-b387-b3912c28537a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kbvcg\" (UID: \"c2d9d246-66bd-40db-b387-b3912c28537a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"
Jul 09 18:11:13.636997 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.636938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2d9d246-66bd-40db-b387-b3912c28537a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kbvcg\" (UID: \"c2d9d246-66bd-40db-b387-b3912c28537a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"
Jul 09 18:11:13.640607 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.640575    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7s5q\" (UniqueName: \"kubernetes.io/projected/c2d9d246-66bd-40db-b387-b3912c28537a-kube-api-access-g7s5q\") pod \"cert-manager-cainjector-559cdd8c87-kbvcg\" (UID: \"c2d9d246-66bd-40db-b387-b3912c28537a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"
Jul 09 18:11:13.746555 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.746519    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"
Jul 09 18:11:13.911697 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:13.911640    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg"]
Jul 09 18:11:13.916259 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:11:13.916136    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2d9d246_66bd_40db_b387_b3912c28537a.slice/crio-6edb2be040734e0edaf214678b301eed389d8994ce4231e860426c904a6b5333 WatchSource:0}: Error finding container 6edb2be040734e0edaf214678b301eed389d8994ce4231e860426c904a6b5333: Status 404 returned error can't find the container with id 6edb2be040734e0edaf214678b301eed389d8994ce4231e860426c904a6b5333
Jul 09 18:11:14.405488 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:14.405444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg" event={"ID":"c2d9d246-66bd-40db-b387-b3912c28537a","Type":"ContainerStarted","Data":"6edb2be040734e0edaf214678b301eed389d8994ce4231e860426c904a6b5333"}
Jul 09 18:11:16.414976 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:16.414936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf" event={"ID":"27388a14-2da8-4e97-85b5-60b7c7e99170","Type":"ContainerStarted","Data":"f71c94d9085c3898ca8e76ae4a3df7d5e46474965e34d214aeaad683358dfeb0"}
Jul 09 18:11:16.415452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:16.415061    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:16.416406 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:16.416380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg" event={"ID":"c2d9d246-66bd-40db-b387-b3912c28537a","Type":"ContainerStarted","Data":"51d8cd089d649b93ecfdc8e6954336b0857d96d2584dfc0e30dc1d8be52aba3a"}
Jul 09 18:11:16.441284 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:16.441117    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf" podStartSLOduration=1.770415251 podStartE2EDuration="5.441093403s" totalImagesPullingTime="3.670678152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:16.441070606 +0000 UTC m=+396.159333371" watchObservedRunningTime="2026-07-09 18:11:16.441093403 +0000 UTC m=+396.159356166"
Jul 09 18:11:16.469723 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:16.469652    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kbvcg" podStartSLOduration=1.612258637 podStartE2EDuration="3.469627807s" totalImagesPullingTime="1.85736917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:16.468534173 +0000 UTC m=+396.186796937" watchObservedRunningTime="2026-07-09 18:11:16.469627807 +0000 UTC m=+396.187890573"
Jul 09 18:11:22.425851 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.425815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7ttrf"
Jul 09 18:11:22.629371 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.629335    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-b9xhj"]
Jul 09 18:11:22.631814 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.631794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-b9xhj"
Jul 09 18:11:22.634607 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.634587    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-twp2f\""
Jul 09 18:11:22.644037 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.644009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-b9xhj"]
Jul 09 18:11:22.802608 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.802574    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08bed58-e734-4c9f-9a8c-ae1394ae065d-bound-sa-token\") pod \"cert-manager-55b4d954d5-b9xhj\" (UID: \"c08bed58-e734-4c9f-9a8c-ae1394ae065d\") " pod="cert-manager/cert-manager-55b4d954d5-b9xhj"
Jul 09 18:11:22.802608 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.802607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96rf7\" (UniqueName: \"kubernetes.io/projected/c08bed58-e734-4c9f-9a8c-ae1394ae065d-kube-api-access-96rf7\") pod \"cert-manager-55b4d954d5-b9xhj\" (UID: \"c08bed58-e734-4c9f-9a8c-ae1394ae065d\") " pod="cert-manager/cert-manager-55b4d954d5-b9xhj"
Jul 09 18:11:22.903983 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.903944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08bed58-e734-4c9f-9a8c-ae1394ae065d-bound-sa-token\") pod \"cert-manager-55b4d954d5-b9xhj\" (UID: \"c08bed58-e734-4c9f-9a8c-ae1394ae065d\") " pod="cert-manager/cert-manager-55b4d954d5-b9xhj"
Jul 09 18:11:22.903983 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.903979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96rf7\" (UniqueName: \"kubernetes.io/projected/c08bed58-e734-4c9f-9a8c-ae1394ae065d-kube-api-access-96rf7\") pod \"cert-manager-55b4d954d5-b9xhj\" (UID: \"c08bed58-e734-4c9f-9a8c-ae1394ae065d\") " pod="cert-manager/cert-manager-55b4d954d5-b9xhj"
Jul 09 18:11:22.913610 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.913584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08bed58-e734-4c9f-9a8c-ae1394ae065d-bound-sa-token\") pod \"cert-manager-55b4d954d5-b9xhj\" (UID: \"c08bed58-e734-4c9f-9a8c-ae1394ae065d\") " pod="cert-manager/cert-manager-55b4d954d5-b9xhj"
Jul 09 18:11:22.914424 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.914406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96rf7\" (UniqueName: \"kubernetes.io/projected/c08bed58-e734-4c9f-9a8c-ae1394ae065d-kube-api-access-96rf7\") pod \"cert-manager-55b4d954d5-b9xhj\" (UID: \"c08bed58-e734-4c9f-9a8c-ae1394ae065d\") " pod="cert-manager/cert-manager-55b4d954d5-b9xhj"
Jul 09 18:11:22.941989 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:22.941958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-b9xhj"
Jul 09 18:11:23.069295 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:23.069270    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-b9xhj"]
Jul 09 18:11:23.071653 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:11:23.071616    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc08bed58_e734_4c9f_9a8c_ae1394ae065d.slice/crio-192965142df1ed90ff8b5d24dbe75f9eb2827b1bd0a8765a30ff924759edc8d0 WatchSource:0}: Error finding container 192965142df1ed90ff8b5d24dbe75f9eb2827b1bd0a8765a30ff924759edc8d0: Status 404 returned error can't find the container with id 192965142df1ed90ff8b5d24dbe75f9eb2827b1bd0a8765a30ff924759edc8d0
Jul 09 18:11:23.445550 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:23.445514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-b9xhj" event={"ID":"c08bed58-e734-4c9f-9a8c-ae1394ae065d","Type":"ContainerStarted","Data":"81e2a144d86bfad6dc472155b946c74d090ea73016a7efe35ad7269fc098c583"}
Jul 09 18:11:23.445550 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:23.445555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-b9xhj" event={"ID":"c08bed58-e734-4c9f-9a8c-ae1394ae065d","Type":"ContainerStarted","Data":"192965142df1ed90ff8b5d24dbe75f9eb2827b1bd0a8765a30ff924759edc8d0"}
Jul 09 18:11:23.472549 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:11:23.472495    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-b9xhj" podStartSLOduration=1.472477643 podStartE2EDuration="1.472477643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:23.472098617 +0000 UTC m=+403.190361381" watchObservedRunningTime="2026-07-09 18:11:23.472477643 +0000 UTC m=+403.190740406"
Jul 09 18:12:08.843855 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.843774    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"]
Jul 09 18:12:08.846080 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.846064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:08.848860 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.848838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 18:12:08.850112 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.850094    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 18:12:08.850234 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.850216    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xndsb\""
Jul 09 18:12:08.850309 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.850223    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 18:12:08.850973 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.850955    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 18:12:08.851022 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.850965    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:12:08.862102 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.862081    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"]
Jul 09 18:12:08.967578 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.967537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f3048f2-55d1-490f-a716-60fa20458deb-cert\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:08.967772 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.967602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f3048f2-55d1-490f-a716-60fa20458deb-metrics-cert\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:08.967772 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.967647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chtfr\" (UniqueName: \"kubernetes.io/projected/2f3048f2-55d1-490f-a716-60fa20458deb-kube-api-access-chtfr\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:08.967772 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:08.967676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f3048f2-55d1-490f-a716-60fa20458deb-manager-config\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.068463 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.068432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f3048f2-55d1-490f-a716-60fa20458deb-cert\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.068654 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.068485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f3048f2-55d1-490f-a716-60fa20458deb-metrics-cert\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.068654 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.068528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtfr\" (UniqueName: \"kubernetes.io/projected/2f3048f2-55d1-490f-a716-60fa20458deb-kube-api-access-chtfr\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.068654 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.068555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f3048f2-55d1-490f-a716-60fa20458deb-manager-config\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.069276 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.069254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f3048f2-55d1-490f-a716-60fa20458deb-manager-config\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.070764 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.070740    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f3048f2-55d1-490f-a716-60fa20458deb-metrics-cert\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.070848 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.070797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f3048f2-55d1-490f-a716-60fa20458deb-cert\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.078685 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.078665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtfr\" (UniqueName: \"kubernetes.io/projected/2f3048f2-55d1-490f-a716-60fa20458deb-kube-api-access-chtfr\") pod \"lws-controller-manager-9fc7df7df-j8gwx\" (UID: \"2f3048f2-55d1-490f-a716-60fa20458deb\") " pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.156102 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.156011    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:09.336025 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.335998    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"]
Jul 09 18:12:09.336294 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:12:09.336270    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3048f2_55d1_490f_a716_60fa20458deb.slice/crio-7f1114c96ac06673f0ce9b307db6ddabec637b9751e0540bdd610eba03c3750e WatchSource:0}: Error finding container 7f1114c96ac06673f0ce9b307db6ddabec637b9751e0540bdd610eba03c3750e: Status 404 returned error can't find the container with id 7f1114c96ac06673f0ce9b307db6ddabec637b9751e0540bdd610eba03c3750e
Jul 09 18:12:09.612253 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:09.612217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx" event={"ID":"2f3048f2-55d1-490f-a716-60fa20458deb","Type":"ContainerStarted","Data":"7f1114c96ac06673f0ce9b307db6ddabec637b9751e0540bdd610eba03c3750e"}
Jul 09 18:12:12.627203 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:12.627112    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx" event={"ID":"2f3048f2-55d1-490f-a716-60fa20458deb","Type":"ContainerStarted","Data":"302108ce15d8ce8dff3a42374fe5ae719c8a8f69aefc1cd5926a577cd3bc40ae"}
Jul 09 18:12:12.627203 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:12.627178    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:12.651439 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:12.651334    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx" podStartSLOduration=1.7477455370000001 podStartE2EDuration="4.651304092s" totalImagesPullingTime="2.903558555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:12.649495974 +0000 UTC m=+452.367758737" watchObservedRunningTime="2026-07-09 18:12:12.651304092 +0000 UTC m=+452.369566857"
Jul 09 18:12:23.633233 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:23.633201    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9fc7df7df-j8gwx"
Jul 09 18:12:43.046155 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.046119    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"]
Jul 09 18:12:43.049217 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.049201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"
Jul 09 18:12:43.051843 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.051823    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tlfdb\""
Jul 09 18:12:43.052837 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.052821    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 18:12:43.052955 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.052879    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 18:12:43.064802 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.064781    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"]
Jul 09 18:12:43.152062 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.152030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rckjm\" (UniqueName: \"kubernetes.io/projected/4495f585-5392-43bf-abba-e323b84280e2-kube-api-access-rckjm\") pod \"limitador-operator-controller-manager-6f9f468797-rpjxx\" (UID: \"4495f585-5392-43bf-abba-e323b84280e2\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"
Jul 09 18:12:43.252488 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.252450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckjm\" (UniqueName: \"kubernetes.io/projected/4495f585-5392-43bf-abba-e323b84280e2-kube-api-access-rckjm\") pod \"limitador-operator-controller-manager-6f9f468797-rpjxx\" (UID: \"4495f585-5392-43bf-abba-e323b84280e2\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"
Jul 09 18:12:43.268967 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.268938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckjm\" (UniqueName: \"kubernetes.io/projected/4495f585-5392-43bf-abba-e323b84280e2-kube-api-access-rckjm\") pod \"limitador-operator-controller-manager-6f9f468797-rpjxx\" (UID: \"4495f585-5392-43bf-abba-e323b84280e2\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"
Jul 09 18:12:43.358666 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.358578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"
Jul 09 18:12:43.549067 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.549011    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"]
Jul 09 18:12:43.551886 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:12:43.551854    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4495f585_5392_43bf_abba_e323b84280e2.slice/crio-11a098360f479e984b27553584c5635624fae8de09eadcbd2b0a99e0447fe4d8 WatchSource:0}: Error finding container 11a098360f479e984b27553584c5635624fae8de09eadcbd2b0a99e0447fe4d8: Status 404 returned error can't find the container with id 11a098360f479e984b27553584c5635624fae8de09eadcbd2b0a99e0447fe4d8
Jul 09 18:12:43.749900 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:43.749865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx" event={"ID":"4495f585-5392-43bf-abba-e323b84280e2","Type":"ContainerStarted","Data":"11a098360f479e984b27553584c5635624fae8de09eadcbd2b0a99e0447fe4d8"}
Jul 09 18:12:47.775410 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:47.775374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx" event={"ID":"4495f585-5392-43bf-abba-e323b84280e2","Type":"ContainerStarted","Data":"d1e1dae54be7ba2a8fbf9744cb28de8792a95a670e5ebfff0b976e75fdc08f91"}
Jul 09 18:12:47.775806 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:47.775448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"
Jul 09 18:12:47.800227 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:47.800171    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx" podStartSLOduration=1.458759025 podStartE2EDuration="4.80014934s" totalImagesPullingTime="3.341390315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:47.79955564 +0000 UTC m=+487.517818423" watchObservedRunningTime="2026-07-09 18:12:47.80014934 +0000 UTC m=+487.518412106"
Jul 09 18:12:58.781078 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:12:58.781043    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-rpjxx"
Jul 09 18:13:42.239362 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.239287    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"]
Jul 09 18:13:42.242271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.242254    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:42.248175 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.248154    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dxql7\""
Jul 09 18:13:42.248792 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.248450    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 18:13:42.269288 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.269259    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"]
Jul 09 18:13:42.330808 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.330775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"]
Jul 09 18:13:42.345828 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.345800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8460e66b-0bf3-438c-b737-0a4cd4b51573-config-file\") pod \"limitador-limitador-ff68ff5cb-s5qd7\" (UID: \"8460e66b-0bf3-438c-b737-0a4cd4b51573\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:42.345987 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.345857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zctkz\" (UniqueName: \"kubernetes.io/projected/8460e66b-0bf3-438c-b737-0a4cd4b51573-kube-api-access-zctkz\") pod \"limitador-limitador-ff68ff5cb-s5qd7\" (UID: \"8460e66b-0bf3-438c-b737-0a4cd4b51573\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:42.447092 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.447060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zctkz\" (UniqueName: \"kubernetes.io/projected/8460e66b-0bf3-438c-b737-0a4cd4b51573-kube-api-access-zctkz\") pod \"limitador-limitador-ff68ff5cb-s5qd7\" (UID: \"8460e66b-0bf3-438c-b737-0a4cd4b51573\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:42.447260 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.447120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8460e66b-0bf3-438c-b737-0a4cd4b51573-config-file\") pod \"limitador-limitador-ff68ff5cb-s5qd7\" (UID: \"8460e66b-0bf3-438c-b737-0a4cd4b51573\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:42.447678 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.447660    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8460e66b-0bf3-438c-b737-0a4cd4b51573-config-file\") pod \"limitador-limitador-ff68ff5cb-s5qd7\" (UID: \"8460e66b-0bf3-438c-b737-0a4cd4b51573\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:42.464252 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.464220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zctkz\" (UniqueName: \"kubernetes.io/projected/8460e66b-0bf3-438c-b737-0a4cd4b51573-kube-api-access-zctkz\") pod \"limitador-limitador-ff68ff5cb-s5qd7\" (UID: \"8460e66b-0bf3-438c-b737-0a4cd4b51573\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:42.552679 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.552589    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:42.702758 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.702638    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"]
Jul 09 18:13:42.705463 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:13:42.705439    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8460e66b_0bf3_438c_b737_0a4cd4b51573.slice/crio-bcd4d54cd05574db0a720422b7a135908e86cd6dfde982ea84781e264b473156 WatchSource:0}: Error finding container bcd4d54cd05574db0a720422b7a135908e86cd6dfde982ea84781e264b473156: Status 404 returned error can't find the container with id bcd4d54cd05574db0a720422b7a135908e86cd6dfde982ea84781e264b473156
Jul 09 18:13:42.998400 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:42.998362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7" event={"ID":"8460e66b-0bf3-438c-b737-0a4cd4b51573","Type":"ContainerStarted","Data":"bcd4d54cd05574db0a720422b7a135908e86cd6dfde982ea84781e264b473156"}
Jul 09 18:13:49.025137 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:49.025100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7" event={"ID":"8460e66b-0bf3-438c-b737-0a4cd4b51573","Type":"ContainerStarted","Data":"ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597"}
Jul 09 18:13:49.025568 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:49.025230    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:58.809647 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:58.809590    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7" podStartSLOduration=11.214479552 podStartE2EDuration="16.809571762s" totalImagesPullingTime="5.59509221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:49.0703497 +0000 UTC m=+548.788612464" watchObservedRunningTime="2026-07-09 18:13:58.809571762 +0000 UTC m=+558.527834526"
Jul 09 18:13:58.810794 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:58.810774    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"]
Jul 09 18:13:58.811040 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:58.811020    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7" podUID="8460e66b-0bf3-438c-b737-0a4cd4b51573" containerName="limitador" containerID="cri-o://ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597" gracePeriod=30
Jul 09 18:13:58.813065 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:58.813042    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:59.359359 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:59.359335    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:13:59.383773 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:59.383676    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zctkz\" (UniqueName: \"kubernetes.io/projected/8460e66b-0bf3-438c-b737-0a4cd4b51573-kube-api-access-zctkz\") pod \"8460e66b-0bf3-438c-b737-0a4cd4b51573\" (UID: \"8460e66b-0bf3-438c-b737-0a4cd4b51573\") "
Jul 09 18:13:59.383773 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:59.383740    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8460e66b-0bf3-438c-b737-0a4cd4b51573-config-file\") pod \"8460e66b-0bf3-438c-b737-0a4cd4b51573\" (UID: \"8460e66b-0bf3-438c-b737-0a4cd4b51573\") "
Jul 09 18:13:59.384372 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:59.384343    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8460e66b-0bf3-438c-b737-0a4cd4b51573-config-file" (OuterVolumeSpecName: "config-file") pod "8460e66b-0bf3-438c-b737-0a4cd4b51573" (UID: "8460e66b-0bf3-438c-b737-0a4cd4b51573"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:13:59.386367 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:59.386340    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8460e66b-0bf3-438c-b737-0a4cd4b51573-kube-api-access-zctkz" (OuterVolumeSpecName: "kube-api-access-zctkz") pod "8460e66b-0bf3-438c-b737-0a4cd4b51573" (UID: "8460e66b-0bf3-438c-b737-0a4cd4b51573"). InnerVolumeSpecName "kube-api-access-zctkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:13:59.484489 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:59.484454    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8460e66b-0bf3-438c-b737-0a4cd4b51573-config-file\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:13:59.484489 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:13:59.484481    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zctkz\" (UniqueName: \"kubernetes.io/projected/8460e66b-0bf3-438c-b737-0a4cd4b51573-kube-api-access-zctkz\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:14:00.070226 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.070189    2565 generic.go:356] "Generic (PLEG): container finished" podID="8460e66b-0bf3-438c-b737-0a4cd4b51573" containerID="ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597" exitCode=0
Jul 09 18:14:00.070226 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.070235    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"
Jul 09 18:14:00.070663 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.070251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7" event={"ID":"8460e66b-0bf3-438c-b737-0a4cd4b51573","Type":"ContainerDied","Data":"ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597"}
Jul 09 18:14:00.070663 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.070306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7" event={"ID":"8460e66b-0bf3-438c-b737-0a4cd4b51573","Type":"ContainerDied","Data":"bcd4d54cd05574db0a720422b7a135908e86cd6dfde982ea84781e264b473156"}
Jul 09 18:14:00.070663 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.070335    2565 scope.go:117] "RemoveContainer" containerID="ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597"
Jul 09 18:14:00.078575 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.078553    2565 scope.go:117] "RemoveContainer" containerID="ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597"
Jul 09 18:14:00.078821 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:14:00.078801    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597\": container with ID starting with ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597 not found: ID does not exist" containerID="ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597"
Jul 09 18:14:00.078884 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.078833    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597"} err="failed to get container status \"ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597\": rpc error: code = NotFound desc = could not find container \"ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597\": container with ID starting with ce04e7eff90d3aaed68cf3b4fc78177afa2b2bc1c48a95bd6e10d003b72e9597 not found: ID does not exist"
Jul 09 18:14:00.094433 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.094408    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"]
Jul 09 18:14:00.099376 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.099352    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-s5qd7"]
Jul 09 18:14:00.868165 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:00.868124    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8460e66b-0bf3-438c-b737-0a4cd4b51573" path="/var/lib/kubelet/pods/8460e66b-0bf3-438c-b737-0a4cd4b51573/volumes"
Jul 09 18:14:40.700205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:40.700172    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:14:40.701002 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:40.700982    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:14:50.012170 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.012133    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-5t6ql"]
Jul 09 18:14:50.012670 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.012650    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8460e66b-0bf3-438c-b737-0a4cd4b51573" containerName="limitador"
Jul 09 18:14:50.012748 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.012673    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8460e66b-0bf3-438c-b737-0a4cd4b51573" containerName="limitador"
Jul 09 18:14:50.012799 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.012752    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8460e66b-0bf3-438c-b737-0a4cd4b51573" containerName="limitador"
Jul 09 18:14:50.015015 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.014993    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:50.017848 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.017827    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:14:50.017975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.017852    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 09 18:14:50.017975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.017828    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:14:50.018841 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.018821    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-ktcpn\""
Jul 09 18:14:50.026586 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.026565    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-p2p26"]
Jul 09 18:14:50.028691 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.028671    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-5t6ql"]
Jul 09 18:14:50.028792 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.028749    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:50.031890 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.031871    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 09 18:14:50.032019 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.031922    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-vwnwz\""
Jul 09 18:14:50.039256 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.039227    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-p2p26"]
Jul 09 18:14:50.113717 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.113685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d00f1c7-bacd-4f5d-a24c-1688ff545add-cert\") pod \"odh-model-controller-84b54b7f97-p2p26\" (UID: \"6d00f1c7-bacd-4f5d-a24c-1688ff545add\") " pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:50.113717 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.113730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2k6j\" (UniqueName: \"kubernetes.io/projected/fe78cc67-8271-4a39-887a-61d35a87e78a-kube-api-access-s2k6j\") pod \"model-serving-api-fd65d7d6-5t6ql\" (UID: \"fe78cc67-8271-4a39-887a-61d35a87e78a\") " pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:50.113969 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.113851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cb4\" (UniqueName: \"kubernetes.io/projected/6d00f1c7-bacd-4f5d-a24c-1688ff545add-kube-api-access-22cb4\") pod \"odh-model-controller-84b54b7f97-p2p26\" (UID: \"6d00f1c7-bacd-4f5d-a24c-1688ff545add\") " pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:50.113969 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.113892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe78cc67-8271-4a39-887a-61d35a87e78a-tls-certs\") pod \"model-serving-api-fd65d7d6-5t6ql\" (UID: \"fe78cc67-8271-4a39-887a-61d35a87e78a\") " pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:50.214975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.214945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cb4\" (UniqueName: \"kubernetes.io/projected/6d00f1c7-bacd-4f5d-a24c-1688ff545add-kube-api-access-22cb4\") pod \"odh-model-controller-84b54b7f97-p2p26\" (UID: \"6d00f1c7-bacd-4f5d-a24c-1688ff545add\") " pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:50.215154 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.214985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe78cc67-8271-4a39-887a-61d35a87e78a-tls-certs\") pod \"model-serving-api-fd65d7d6-5t6ql\" (UID: \"fe78cc67-8271-4a39-887a-61d35a87e78a\") " pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:50.215154 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.215017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d00f1c7-bacd-4f5d-a24c-1688ff545add-cert\") pod \"odh-model-controller-84b54b7f97-p2p26\" (UID: \"6d00f1c7-bacd-4f5d-a24c-1688ff545add\") " pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:50.215154 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.215034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2k6j\" (UniqueName: \"kubernetes.io/projected/fe78cc67-8271-4a39-887a-61d35a87e78a-kube-api-access-s2k6j\") pod \"model-serving-api-fd65d7d6-5t6ql\" (UID: \"fe78cc67-8271-4a39-887a-61d35a87e78a\") " pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:50.215154 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:14:50.215102    2565 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 09 18:14:50.215357 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:14:50.215170    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe78cc67-8271-4a39-887a-61d35a87e78a-tls-certs podName:fe78cc67-8271-4a39-887a-61d35a87e78a nodeName:}" failed. No retries permitted until 2026-07-09 18:14:50.715153437 +0000 UTC m=+610.433416178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe78cc67-8271-4a39-887a-61d35a87e78a-tls-certs") pod "model-serving-api-fd65d7d6-5t6ql" (UID: "fe78cc67-8271-4a39-887a-61d35a87e78a") : secret "model-serving-api-tls" not found
Jul 09 18:14:50.215357 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:14:50.215108    2565 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 18:14:50.215357 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:14:50.215259    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d00f1c7-bacd-4f5d-a24c-1688ff545add-cert podName:6d00f1c7-bacd-4f5d-a24c-1688ff545add nodeName:}" failed. No retries permitted until 2026-07-09 18:14:50.715239385 +0000 UTC m=+610.433502140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6d00f1c7-bacd-4f5d-a24c-1688ff545add-cert") pod "odh-model-controller-84b54b7f97-p2p26" (UID: "6d00f1c7-bacd-4f5d-a24c-1688ff545add") : secret "odh-model-controller-webhook-cert" not found
Jul 09 18:14:50.224042 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.224010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2k6j\" (UniqueName: \"kubernetes.io/projected/fe78cc67-8271-4a39-887a-61d35a87e78a-kube-api-access-s2k6j\") pod \"model-serving-api-fd65d7d6-5t6ql\" (UID: \"fe78cc67-8271-4a39-887a-61d35a87e78a\") " pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:50.224159 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.224135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cb4\" (UniqueName: \"kubernetes.io/projected/6d00f1c7-bacd-4f5d-a24c-1688ff545add-kube-api-access-22cb4\") pod \"odh-model-controller-84b54b7f97-p2p26\" (UID: \"6d00f1c7-bacd-4f5d-a24c-1688ff545add\") " pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:50.719502 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.719464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe78cc67-8271-4a39-887a-61d35a87e78a-tls-certs\") pod \"model-serving-api-fd65d7d6-5t6ql\" (UID: \"fe78cc67-8271-4a39-887a-61d35a87e78a\") " pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:50.719746 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.719517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d00f1c7-bacd-4f5d-a24c-1688ff545add-cert\") pod \"odh-model-controller-84b54b7f97-p2p26\" (UID: \"6d00f1c7-bacd-4f5d-a24c-1688ff545add\") " pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:50.721898 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.721868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe78cc67-8271-4a39-887a-61d35a87e78a-tls-certs\") pod \"model-serving-api-fd65d7d6-5t6ql\" (UID: \"fe78cc67-8271-4a39-887a-61d35a87e78a\") " pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:50.722037 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.722022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d00f1c7-bacd-4f5d-a24c-1688ff545add-cert\") pod \"odh-model-controller-84b54b7f97-p2p26\" (UID: \"6d00f1c7-bacd-4f5d-a24c-1688ff545add\") " pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:50.924614 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.924584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:50.937755 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:50.937725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:51.064622 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:51.064593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-5t6ql"]
Jul 09 18:14:51.067294 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:14:51.067256    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe78cc67_8271_4a39_887a_61d35a87e78a.slice/crio-66d67b2b12290cb754215c5d227b935e226121d6a31b6db726594206365dfca4 WatchSource:0}: Error finding container 66d67b2b12290cb754215c5d227b935e226121d6a31b6db726594206365dfca4: Status 404 returned error can't find the container with id 66d67b2b12290cb754215c5d227b935e226121d6a31b6db726594206365dfca4
Jul 09 18:14:51.086264 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:51.086244    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-p2p26"]
Jul 09 18:14:51.088487 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:14:51.088463    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d00f1c7_bacd_4f5d_a24c_1688ff545add.slice/crio-158fe844aca35c93a1501591709b3280be49b5316f0e741535d798af4d927411 WatchSource:0}: Error finding container 158fe844aca35c93a1501591709b3280be49b5316f0e741535d798af4d927411: Status 404 returned error can't find the container with id 158fe844aca35c93a1501591709b3280be49b5316f0e741535d798af4d927411
Jul 09 18:14:51.267593 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:51.267489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-5t6ql" event={"ID":"fe78cc67-8271-4a39-887a-61d35a87e78a","Type":"ContainerStarted","Data":"66d67b2b12290cb754215c5d227b935e226121d6a31b6db726594206365dfca4"}
Jul 09 18:14:51.268557 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:51.268526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-p2p26" event={"ID":"6d00f1c7-bacd-4f5d-a24c-1688ff545add","Type":"ContainerStarted","Data":"158fe844aca35c93a1501591709b3280be49b5316f0e741535d798af4d927411"}
Jul 09 18:14:53.282114 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:53.282073    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-5t6ql" event={"ID":"fe78cc67-8271-4a39-887a-61d35a87e78a","Type":"ContainerStarted","Data":"9303e9781e72e8e4820c5dc82f83068b1111db34daa823d29b4917c5d98f34a6"}
Jul 09 18:14:53.282547 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:53.282360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:14:53.302000 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:53.301943    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-5t6ql" podStartSLOduration=2.704792531 podStartE2EDuration="4.301924027s" totalImagesPullingTime="1.597131496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:53.300796977 +0000 UTC m=+613.019059744" watchObservedRunningTime="2026-07-09 18:14:53.301924027 +0000 UTC m=+613.020186787"
Jul 09 18:14:54.287709 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:54.287666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-p2p26" event={"ID":"6d00f1c7-bacd-4f5d-a24c-1688ff545add","Type":"ContainerStarted","Data":"bb7aacc9e16031bf45a158684e0022097ec78b67e59956fa35d8eedd2ea40e2e"}
Jul 09 18:14:54.288122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:54.287927    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:14:54.308472 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:14:54.308422    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-p2p26" podStartSLOduration=1.264173284 podStartE2EDuration="4.308401396s" totalImagesPullingTime="3.044228112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:54.306812157 +0000 UTC m=+614.025074920" watchObservedRunningTime="2026-07-09 18:14:54.308401396 +0000 UTC m=+614.026664217"
Jul 09 18:15:04.291891 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:04.291813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-5t6ql"
Jul 09 18:15:05.294634 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:05.294604    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-p2p26"
Jul 09 18:15:06.277454 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.277411    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-55kmp"]
Jul 09 18:15:06.280819 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.280793    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-55kmp"
Jul 09 18:15:06.283715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.283676    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 18:15:06.283715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.283707    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tccgw\""
Jul 09 18:15:06.291899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.291871    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-55kmp"]
Jul 09 18:15:06.356130 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.356093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzclc\" (UniqueName: \"kubernetes.io/projected/9a57269e-81f9-4d91-9e59-05efa447d487-kube-api-access-mzclc\") pod \"s3-init-55kmp\" (UID: \"9a57269e-81f9-4d91-9e59-05efa447d487\") " pod="kserve/s3-init-55kmp"
Jul 09 18:15:06.356507 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.356175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a57269e-81f9-4d91-9e59-05efa447d487-model-cache\") pod \"s3-init-55kmp\" (UID: \"9a57269e-81f9-4d91-9e59-05efa447d487\") " pod="kserve/s3-init-55kmp"
Jul 09 18:15:06.457138 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.457101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzclc\" (UniqueName: \"kubernetes.io/projected/9a57269e-81f9-4d91-9e59-05efa447d487-kube-api-access-mzclc\") pod \"s3-init-55kmp\" (UID: \"9a57269e-81f9-4d91-9e59-05efa447d487\") " pod="kserve/s3-init-55kmp"
Jul 09 18:15:06.457290 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.457163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a57269e-81f9-4d91-9e59-05efa447d487-model-cache\") pod \"s3-init-55kmp\" (UID: \"9a57269e-81f9-4d91-9e59-05efa447d487\") " pod="kserve/s3-init-55kmp"
Jul 09 18:15:06.457513 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.457497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a57269e-81f9-4d91-9e59-05efa447d487-model-cache\") pod \"s3-init-55kmp\" (UID: \"9a57269e-81f9-4d91-9e59-05efa447d487\") " pod="kserve/s3-init-55kmp"
Jul 09 18:15:06.467335 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.467312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzclc\" (UniqueName: \"kubernetes.io/projected/9a57269e-81f9-4d91-9e59-05efa447d487-kube-api-access-mzclc\") pod \"s3-init-55kmp\" (UID: \"9a57269e-81f9-4d91-9e59-05efa447d487\") " pod="kserve/s3-init-55kmp"
Jul 09 18:15:06.590352 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.590253    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-55kmp"
Jul 09 18:15:06.724449 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:06.724411    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-55kmp"]
Jul 09 18:15:06.728417 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:15:06.728391    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a57269e_81f9_4d91_9e59_05efa447d487.slice/crio-a3ab0758e7ef975846177427252d9313d66810e630566d5d24dfea85a9446d7a WatchSource:0}: Error finding container a3ab0758e7ef975846177427252d9313d66810e630566d5d24dfea85a9446d7a: Status 404 returned error can't find the container with id a3ab0758e7ef975846177427252d9313d66810e630566d5d24dfea85a9446d7a
Jul 09 18:15:07.341704 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:07.341655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-55kmp" event={"ID":"9a57269e-81f9-4d91-9e59-05efa447d487","Type":"ContainerStarted","Data":"a3ab0758e7ef975846177427252d9313d66810e630566d5d24dfea85a9446d7a"}
Jul 09 18:15:12.362917 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:12.362876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-55kmp" event={"ID":"9a57269e-81f9-4d91-9e59-05efa447d487","Type":"ContainerStarted","Data":"c567ce9116a17ce67170614ba168a1d2d543aaae2397c72ea77641890d95ad0a"}
Jul 09 18:15:13.367987 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:13.367885    2565 generic.go:356] "Generic (PLEG): container finished" podID="9a57269e-81f9-4d91-9e59-05efa447d487" containerID="c567ce9116a17ce67170614ba168a1d2d543aaae2397c72ea77641890d95ad0a" exitCode=0
Jul 09 18:15:13.367987 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:13.367960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-55kmp" event={"ID":"9a57269e-81f9-4d91-9e59-05efa447d487","Type":"ContainerDied","Data":"c567ce9116a17ce67170614ba168a1d2d543aaae2397c72ea77641890d95ad0a"}
Jul 09 18:15:18.396206 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:18.396151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-55kmp" event={"ID":"9a57269e-81f9-4d91-9e59-05efa447d487","Type":"ContainerStarted","Data":"4e3028b3b7719052077a03801d7a2a4cbb3f50589a058362faeb159d0054ae74"}
Jul 09 18:15:18.416190 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:15:18.416128    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-55kmp" podStartSLOduration=1.230911717 podStartE2EDuration="12.416112589s" totalImagesPullingTime="9.831170478s" totalInitContainerRuntime="1.354030394s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:18.415343124 +0000 UTC m=+638.133605889" watchObservedRunningTime="2026-07-09 18:15:18.416112589 +0000 UTC m=+638.134375354"
Jul 09 18:16:23.660414 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:23.660379    2565 generic.go:356] "Generic (PLEG): container finished" podID="9a57269e-81f9-4d91-9e59-05efa447d487" containerID="4e3028b3b7719052077a03801d7a2a4cbb3f50589a058362faeb159d0054ae74" exitCode=0
Jul 09 18:16:23.660414 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:23.660417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-55kmp" event={"ID":"9a57269e-81f9-4d91-9e59-05efa447d487","Type":"ContainerDied","Data":"4e3028b3b7719052077a03801d7a2a4cbb3f50589a058362faeb159d0054ae74"}
Jul 09 18:16:24.790651 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:24.790624    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-55kmp"
Jul 09 18:16:24.935234 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:24.935149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a57269e-81f9-4d91-9e59-05efa447d487-model-cache\") pod \"9a57269e-81f9-4d91-9e59-05efa447d487\" (UID: \"9a57269e-81f9-4d91-9e59-05efa447d487\") "
Jul 09 18:16:24.935234 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:24.935207    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzclc\" (UniqueName: \"kubernetes.io/projected/9a57269e-81f9-4d91-9e59-05efa447d487-kube-api-access-mzclc\") pod \"9a57269e-81f9-4d91-9e59-05efa447d487\" (UID: \"9a57269e-81f9-4d91-9e59-05efa447d487\") "
Jul 09 18:16:24.937450 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:24.937423    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a57269e-81f9-4d91-9e59-05efa447d487-kube-api-access-mzclc" (OuterVolumeSpecName: "kube-api-access-mzclc") pod "9a57269e-81f9-4d91-9e59-05efa447d487" (UID: "9a57269e-81f9-4d91-9e59-05efa447d487"). InnerVolumeSpecName "kube-api-access-mzclc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:16:24.954098 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:24.954071    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a57269e-81f9-4d91-9e59-05efa447d487-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a57269e-81f9-4d91-9e59-05efa447d487" (UID: "9a57269e-81f9-4d91-9e59-05efa447d487"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:25.036035 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:25.036008    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzclc\" (UniqueName: \"kubernetes.io/projected/9a57269e-81f9-4d91-9e59-05efa447d487-kube-api-access-mzclc\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:25.036035 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:25.036033    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a57269e-81f9-4d91-9e59-05efa447d487-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:25.669420 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:25.669392    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-55kmp"
Jul 09 18:16:25.669420 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:25.669403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-55kmp" event={"ID":"9a57269e-81f9-4d91-9e59-05efa447d487","Type":"ContainerDied","Data":"a3ab0758e7ef975846177427252d9313d66810e630566d5d24dfea85a9446d7a"}
Jul 09 18:16:25.669617 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:25.669435    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ab0758e7ef975846177427252d9313d66810e630566d5d24dfea85a9446d7a"
Jul 09 18:16:38.707680 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.707488    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"]
Jul 09 18:16:38.708494 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.708465    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a57269e-81f9-4d91-9e59-05efa447d487" containerName="download-hf-model"
Jul 09 18:16:38.708494 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.708494    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a57269e-81f9-4d91-9e59-05efa447d487" containerName="download-hf-model"
Jul 09 18:16:38.708682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.708532    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a57269e-81f9-4d91-9e59-05efa447d487" containerName="s3-init"
Jul 09 18:16:38.708682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.708541    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a57269e-81f9-4d91-9e59-05efa447d487" containerName="s3-init"
Jul 09 18:16:38.708682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.708624    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a57269e-81f9-4d91-9e59-05efa447d487" containerName="s3-init"
Jul 09 18:16:38.710785 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.710768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.714327 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.714306    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:16:38.714454 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.714345    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:16:38.714454 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.714313    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:16:38.714568 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.714478    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-7sv49\""
Jul 09 18:16:38.724706 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.724681    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"]
Jul 09 18:16:38.753289 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.753251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.753545 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.753525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.753733 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.753716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.753846 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.753831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.753977 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.753963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.754161 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.754117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.754276 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.754249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.754346 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.754299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.754402 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.754383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs6h4\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-kube-api-access-fs6h4\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.855730 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.855696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.855730 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.855734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856042 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.855766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856042 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.855807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856042 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.855849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856042 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.855873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856042 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.855895    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs6h4\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-kube-api-access-fs6h4\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856042 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.855991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856042 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.856017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856393 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.856370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856567 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.856541    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856647 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.856558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856647 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.856622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.856719 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.856658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.858367 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.858346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.858605 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.858584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.873631 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.873598    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:38.877003 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:38.876976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs6h4\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-kube-api-access-fs6h4\") pod \"router-gateway-1-openshift-default-799f46c59b-vp6np\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:39.024968 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:39.024850    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:39.163295 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:39.163270    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"]
Jul 09 18:16:39.165815 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:16:39.165786    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ac22c8_bb90_492b_a98d_f6306d90ad08.slice/crio-55690a76e842aa1b826cb09d47480ec5827a7126530cb0d4d65d8d6504bc1729 WatchSource:0}: Error finding container 55690a76e842aa1b826cb09d47480ec5827a7126530cb0d4d65d8d6504bc1729: Status 404 returned error can't find the container with id 55690a76e842aa1b826cb09d47480ec5827a7126530cb0d4d65d8d6504bc1729
Jul 09 18:16:39.167714 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:39.167698    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:16:39.729821 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:39.729788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np" event={"ID":"a6ac22c8-bb90-492b-a98d-f6306d90ad08","Type":"ContainerStarted","Data":"55690a76e842aa1b826cb09d47480ec5827a7126530cb0d4d65d8d6504bc1729"}
Jul 09 18:16:40.720126 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:40.720096    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"]
Jul 09 18:16:42.716876 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:42.716835    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:16:42.717151 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:42.716950    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:16:42.717151 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:42.716988    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:16:43.753872 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:43.753832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np" event={"ID":"a6ac22c8-bb90-492b-a98d-f6306d90ad08","Type":"ContainerStarted","Data":"f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f"}
Jul 09 18:16:43.754320 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:43.753928    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np" podUID="a6ac22c8-bb90-492b-a98d-f6306d90ad08" containerName="istio-proxy" containerID="cri-o://f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f" gracePeriod=30
Jul 09 18:16:43.778998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:43.778934    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np" podStartSLOduration=2.230134709 podStartE2EDuration="5.778900254s" totalImagesPullingTime="3.548765545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:43.775677128 +0000 UTC m=+723.493939882" watchObservedRunningTime="2026-07-09 18:16:43.778900254 +0000 UTC m=+723.497163019"
Jul 09 18:16:44.169365 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.169268    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"]
Jul 09 18:16:44.172811 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.172791    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.175623 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.175598    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s8z92\""
Jul 09 18:16:44.175760 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.175663    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 09 18:16:44.184063 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.184040    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"]
Jul 09 18:16:44.202389 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.202351    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.202618 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.202476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.202618 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.202505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.202618 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.202531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6d380f-dafd-4263-b018-301261a1e223-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.202618 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.202568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56dqt\" (UniqueName: \"kubernetes.io/projected/ab6d380f-dafd-4263-b018-301261a1e223-kube-api-access-56dqt\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.202836 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.202632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.304036 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.304007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.304036 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.304039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.304280 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.304058    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6d380f-dafd-4263-b018-301261a1e223-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.304280 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.304088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56dqt\" (UniqueName: \"kubernetes.io/projected/ab6d380f-dafd-4263-b018-301261a1e223-kube-api-access-56dqt\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.304280 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.304143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.304280 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.304194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.304566 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.304532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.304685 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.304661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.304817 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.304797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.307120 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.307097    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.307258 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.307241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6d380f-dafd-4263-b018-301261a1e223-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.312714 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.312690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56dqt\" (UniqueName: \"kubernetes.io/projected/ab6d380f-dafd-4263-b018-301261a1e223-kube-api-access-56dqt\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.484052 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.484017    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:44.619365 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.619267    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"]
Jul 09 18:16:44.622278 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:16:44.622244    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6d380f_dafd_4263_b018_301261a1e223.slice/crio-cf8cf730e70abb7509d9d856945a0a2c27465d8f3ed2ad68215293d154dc0142 WatchSource:0}: Error finding container cf8cf730e70abb7509d9d856945a0a2c27465d8f3ed2ad68215293d154dc0142: Status 404 returned error can't find the container with id cf8cf730e70abb7509d9d856945a0a2c27465d8f3ed2ad68215293d154dc0142
Jul 09 18:16:44.759692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:44.759604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj" event={"ID":"ab6d380f-dafd-4263-b018-301261a1e223","Type":"ContainerStarted","Data":"cf8cf730e70abb7509d9d856945a0a2c27465d8f3ed2ad68215293d154dc0142"}
Jul 09 18:16:46.771729 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:46.771691    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj" event={"ID":"ab6d380f-dafd-4263-b018-301261a1e223","Type":"ContainerStarted","Data":"4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b"}
Jul 09 18:16:46.794621 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:46.794553    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj" podStartSLOduration=1.39470696 podStartE2EDuration="2.794530927s" totalImagesPullingTime="1.399823967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:46.791926641 +0000 UTC m=+726.510189405" watchObservedRunningTime="2026-07-09 18:16:46.794530927 +0000 UTC m=+726.512793691"
Jul 09 18:16:47.484601 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:47.484562    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:49.008563 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.008540    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:49.050063 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.049979    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-data\") pod \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") "
Jul 09 18:16:49.050063 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050041    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-token\") pod \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") "
Jul 09 18:16:49.050274 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050086    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istiod-ca-cert\") pod \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") "
Jul 09 18:16:49.050274 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050119    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-podinfo\") pod \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") "
Jul 09 18:16:49.050274 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-socket\") pod \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") "
Jul 09 18:16:49.050274 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050177    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-credential-socket\") pod \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") "
Jul 09 18:16:49.050274 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050209    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs6h4\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-kube-api-access-fs6h4\") pod \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") "
Jul 09 18:16:49.050274 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050257    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-envoy\") pod \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") "
Jul 09 18:16:49.050274 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050260    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-data" (OuterVolumeSpecName: "istio-data") pod "a6ac22c8-bb90-492b-a98d-f6306d90ad08" (UID: "a6ac22c8-bb90-492b-a98d-f6306d90ad08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:49.050585 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050280    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-certs\") pod \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\" (UID: \"a6ac22c8-bb90-492b-a98d-f6306d90ad08\") "
Jul 09 18:16:49.050585 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050531    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a6ac22c8-bb90-492b-a98d-f6306d90ad08" (UID: "a6ac22c8-bb90-492b-a98d-f6306d90ad08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:49.050715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050599    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-data\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:49.050715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050626    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a6ac22c8-bb90-492b-a98d-f6306d90ad08" (UID: "a6ac22c8-bb90-492b-a98d-f6306d90ad08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:49.050800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a6ac22c8-bb90-492b-a98d-f6306d90ad08" (UID: "a6ac22c8-bb90-492b-a98d-f6306d90ad08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:16:49.050983 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.050942    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a6ac22c8-bb90-492b-a98d-f6306d90ad08" (UID: "a6ac22c8-bb90-492b-a98d-f6306d90ad08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:49.052535 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.052509    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-token" (OuterVolumeSpecName: "istio-token") pod "a6ac22c8-bb90-492b-a98d-f6306d90ad08" (UID: "a6ac22c8-bb90-492b-a98d-f6306d90ad08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:16:49.052641 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.052628    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a6ac22c8-bb90-492b-a98d-f6306d90ad08" (UID: "a6ac22c8-bb90-492b-a98d-f6306d90ad08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:49.052828 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.052797    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-kube-api-access-fs6h4" (OuterVolumeSpecName: "kube-api-access-fs6h4") pod "a6ac22c8-bb90-492b-a98d-f6306d90ad08" (UID: "a6ac22c8-bb90-492b-a98d-f6306d90ad08"). InnerVolumeSpecName "kube-api-access-fs6h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:16:49.052939 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.052867    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a6ac22c8-bb90-492b-a98d-f6306d90ad08" (UID: "a6ac22c8-bb90-492b-a98d-f6306d90ad08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 18:16:49.151423 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.151392    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-podinfo\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:49.151423 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.151421    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:49.151423 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.151430    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-credential-socket\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:49.151648 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.151439    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs6h4\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-kube-api-access-fs6h4\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:49.151648 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.151448    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-envoy\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:49.151648 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.151456    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6ac22c8-bb90-492b-a98d-f6306d90ad08-workload-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:49.151648 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.151465    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istio-token\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:49.151648 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.151472    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6ac22c8-bb90-492b-a98d-f6306d90ad08-istiod-ca-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:16:49.790062 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.790023    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6ac22c8-bb90-492b-a98d-f6306d90ad08" containerID="f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f" exitCode=0
Jul 09 18:16:49.790286 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.790072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np" event={"ID":"a6ac22c8-bb90-492b-a98d-f6306d90ad08","Type":"ContainerDied","Data":"f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f"}
Jul 09 18:16:49.790286 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.790079    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"
Jul 09 18:16:49.790286 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.790150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np" event={"ID":"a6ac22c8-bb90-492b-a98d-f6306d90ad08","Type":"ContainerDied","Data":"55690a76e842aa1b826cb09d47480ec5827a7126530cb0d4d65d8d6504bc1729"}
Jul 09 18:16:49.790286 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.790175    2565 scope.go:117] "RemoveContainer" containerID="f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f"
Jul 09 18:16:49.799342 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.799324    2565 scope.go:117] "RemoveContainer" containerID="f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f"
Jul 09 18:16:49.799583 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:16:49.799561    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f\": container with ID starting with f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f not found: ID does not exist" containerID="f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f"
Jul 09 18:16:49.799674 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.799589    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f"} err="failed to get container status \"f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f\": rpc error: code = NotFound desc = could not find container \"f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f\": container with ID starting with f5b537c3f4830011460293a97cebe87a6e56c60dbb1529691565ec715906e98f not found: ID does not exist"
Jul 09 18:16:49.815646 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.815621    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"]
Jul 09 18:16:49.820565 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:49.820541    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np"]
Jul 09 18:16:50.867986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:50.867955    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6ac22c8-bb90-492b-a98d-f6306d90ad08" path="/var/lib/kubelet/pods/a6ac22c8-bb90-492b-a98d-f6306d90ad08/volumes"
Jul 09 18:16:54.484777 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:54.484740    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:54.497642 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:54.497613    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:16:54.825080 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:16:54.825000    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:17:12.869481 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.869452    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"]
Jul 09 18:17:12.869845 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.869781    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ac22c8-bb90-492b-a98d-f6306d90ad08" containerName="istio-proxy"
Jul 09 18:17:12.869845 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.869793    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ac22c8-bb90-492b-a98d-f6306d90ad08" containerName="istio-proxy"
Jul 09 18:17:12.869934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.869875    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ac22c8-bb90-492b-a98d-f6306d90ad08" containerName="istio-proxy"
Jul 09 18:17:12.872894 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.872877    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:12.875655 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.875631    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 09 18:17:12.875775 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.875669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:17:12.885501 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.885476    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"]
Jul 09 18:17:12.954653 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.954625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-dshm\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:12.954856 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.954669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:12.954856 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.954820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-home\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:12.955001 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.954858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b313bdf0-269c-4a97-b216-ded35085a7f5-tls-certs\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:12.955063 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.955037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:12.955118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.955102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-model-cache\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:12.955186 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:12.955157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4lc\" (UniqueName: \"kubernetes.io/projected/b313bdf0-269c-4a97-b216-ded35085a7f5-kube-api-access-cm4lc\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057040 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-home\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057040 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b313bdf0-269c-4a97-b216-ded35085a7f5-tls-certs\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057285 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057082    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057285 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-model-cache\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057285 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4lc\" (UniqueName: \"kubernetes.io/projected/b313bdf0-269c-4a97-b216-ded35085a7f5-kube-api-access-cm4lc\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057285 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-dshm\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057285 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057545 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057609 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-home\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057609 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-model-cache\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.057683 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.057659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.059480 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.059461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-dshm\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.059619 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.059602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b313bdf0-269c-4a97-b216-ded35085a7f5-tls-certs\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.066349 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.066328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4lc\" (UniqueName: \"kubernetes.io/projected/b313bdf0-269c-4a97-b216-ded35085a7f5-kube-api-access-cm4lc\") pod \"precise-prefix-cache-test-kserve-7c4c648d54-2296z\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.183857 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.183754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:13.260851 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.260818    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"]
Jul 09 18:17:13.272829 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.272802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.275704 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.275666    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-wktvg\""
Jul 09 18:17:13.281434 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.281200    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"]
Jul 09 18:17:13.331776 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.331744    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"]
Jul 09 18:17:13.335296 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:17:13.335265    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb313bdf0_269c_4a97_b216_ded35085a7f5.slice/crio-6d8ebb82f527dcbc865bec7a6211292ef4b18704d7d53f9cc8065574f6da67ab WatchSource:0}: Error finding container 6d8ebb82f527dcbc865bec7a6211292ef4b18704d7d53f9cc8065574f6da67ab: Status 404 returned error can't find the container with id 6d8ebb82f527dcbc865bec7a6211292ef4b18704d7d53f9cc8065574f6da67ab
Jul 09 18:17:13.359672 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.359630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.359788 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.359675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.359788 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.359701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.359788 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.359771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.359895 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.359815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.359955 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.359940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqcc\" (UniqueName: \"kubernetes.io/projected/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kube-api-access-9dqcc\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.460784 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.460746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqcc\" (UniqueName: \"kubernetes.io/projected/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kube-api-access-9dqcc\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.460784 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.460799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.461074 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.460819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.461074 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.460836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.461074 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.460857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.461074 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.460882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.461294 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.461217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.461294 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.461252    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.461401 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.461298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.461401 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.461347    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.463270 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.463246    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.469968 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.469939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqcc\" (UniqueName: \"kubernetes.io/projected/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kube-api-access-9dqcc\") pod \"precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.584946 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.584890    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:13.730353 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.730327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"]
Jul 09 18:17:13.733256 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:17:13.733213    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd158fd4_7340_4858_b9d9_99c29c4fc6cb.slice/crio-24ed213fec9e226f0250667150bb52edc8d01a080e6e0697f92dae75b5eea34e WatchSource:0}: Error finding container 24ed213fec9e226f0250667150bb52edc8d01a080e6e0697f92dae75b5eea34e: Status 404 returned error can't find the container with id 24ed213fec9e226f0250667150bb52edc8d01a080e6e0697f92dae75b5eea34e
Jul 09 18:17:13.908872 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.908833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z" event={"ID":"b313bdf0-269c-4a97-b216-ded35085a7f5","Type":"ContainerStarted","Data":"f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7"}
Jul 09 18:17:13.909342 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.908917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z" event={"ID":"b313bdf0-269c-4a97-b216-ded35085a7f5","Type":"ContainerStarted","Data":"6d8ebb82f527dcbc865bec7a6211292ef4b18704d7d53f9cc8065574f6da67ab"}
Jul 09 18:17:13.910505 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.910480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerStarted","Data":"819a0f82350e3a55c095f3884abd36be109bd95554498caeaa28ffd9592f10d2"}
Jul 09 18:17:13.910641 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:13.910529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerStarted","Data":"24ed213fec9e226f0250667150bb52edc8d01a080e6e0697f92dae75b5eea34e"}
Jul 09 18:17:14.917594 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:14.917501    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerID="819a0f82350e3a55c095f3884abd36be109bd95554498caeaa28ffd9592f10d2" exitCode=0
Jul 09 18:17:14.917594 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:14.917579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerDied","Data":"819a0f82350e3a55c095f3884abd36be109bd95554498caeaa28ffd9592f10d2"}
Jul 09 18:17:15.882284 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:15.881514    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"]
Jul 09 18:17:16.932868 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:16.932833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerStarted","Data":"404901fd2c81070464238bfeec899ebc8d415a4c4ad463cf5878cf80b493b934"}
Jul 09 18:17:17.941286 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:17.941239    2565 generic.go:356] "Generic (PLEG): container finished" podID="b313bdf0-269c-4a97-b216-ded35085a7f5" containerID="f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7" exitCode=0
Jul 09 18:17:17.941741 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:17.941291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z" event={"ID":"b313bdf0-269c-4a97-b216-ded35085a7f5","Type":"ContainerDied","Data":"f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7"}
Jul 09 18:17:18.948888 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:18.948852    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z" event={"ID":"b313bdf0-269c-4a97-b216-ded35085a7f5","Type":"ContainerStarted","Data":"21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8"}
Jul 09 18:17:18.970012 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:18.969965    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z" podStartSLOduration=3.154965012 podStartE2EDuration="6.969947565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.814982553s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:18.969577642 +0000 UTC m=+758.687840405" watchObservedRunningTime="2026-07-09 18:17:18.969947565 +0000 UTC m=+758.688210333"
Jul 09 18:17:19.184166 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:19.184123    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:23.184760 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:23.184720    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:23.199899 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:23.199871    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:23.991349 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:23.991318    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:17:26.993356 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:26.993323    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj_cd158fd4-7340-4858-b9d9-99c29c4fc6cb/main/0.log"
Jul 09 18:17:26.993810 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:26.993751    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerID="404901fd2c81070464238bfeec899ebc8d415a4c4ad463cf5878cf80b493b934" exitCode=1
Jul 09 18:17:26.993877 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:26.993827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerDied","Data":"404901fd2c81070464238bfeec899ebc8d415a4c4ad463cf5878cf80b493b934"}
Jul 09 18:17:30.907172 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:30.907123    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj" podUID="ab6d380f-dafd-4263-b018-301261a1e223" containerName="main" containerID="cri-o://4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b" gracePeriod=45
Jul 09 18:17:31.199825 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.199789    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:17:31.228378 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.228339    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-tmp-dir\") pod \"ab6d380f-dafd-4263-b018-301261a1e223\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") "
Jul 09 18:17:31.228564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.228389    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-dshm\") pod \"ab6d380f-dafd-4263-b018-301261a1e223\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") "
Jul 09 18:17:31.228564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.228412    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-home\") pod \"ab6d380f-dafd-4263-b018-301261a1e223\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") "
Jul 09 18:17:31.228564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.228436    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6d380f-dafd-4263-b018-301261a1e223-tls-certs\") pod \"ab6d380f-dafd-4263-b018-301261a1e223\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") "
Jul 09 18:17:31.228564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.228473    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56dqt\" (UniqueName: \"kubernetes.io/projected/ab6d380f-dafd-4263-b018-301261a1e223-kube-api-access-56dqt\") pod \"ab6d380f-dafd-4263-b018-301261a1e223\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") "
Jul 09 18:17:31.228564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.228530    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-model-cache\") pod \"ab6d380f-dafd-4263-b018-301261a1e223\" (UID: \"ab6d380f-dafd-4263-b018-301261a1e223\") "
Jul 09 18:17:31.228826 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.228660    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab6d380f-dafd-4263-b018-301261a1e223" (UID: "ab6d380f-dafd-4263-b018-301261a1e223"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:31.228955 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.228928    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-home" (OuterVolumeSpecName: "home") pod "ab6d380f-dafd-4263-b018-301261a1e223" (UID: "ab6d380f-dafd-4263-b018-301261a1e223"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:31.229258 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.229207    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.229258 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.229233    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.229441 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.229301    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab6d380f-dafd-4263-b018-301261a1e223" (UID: "ab6d380f-dafd-4263-b018-301261a1e223"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:31.231168 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.231139    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6d380f-dafd-4263-b018-301261a1e223-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab6d380f-dafd-4263-b018-301261a1e223" (UID: "ab6d380f-dafd-4263-b018-301261a1e223"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:17:31.231290 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.231260    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6d380f-dafd-4263-b018-301261a1e223-kube-api-access-56dqt" (OuterVolumeSpecName: "kube-api-access-56dqt") pod "ab6d380f-dafd-4263-b018-301261a1e223" (UID: "ab6d380f-dafd-4263-b018-301261a1e223"). InnerVolumeSpecName "kube-api-access-56dqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:31.231420 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.231399    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-dshm" (OuterVolumeSpecName: "dshm") pod "ab6d380f-dafd-4263-b018-301261a1e223" (UID: "ab6d380f-dafd-4263-b018-301261a1e223"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:31.330498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.330456    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.330498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.330492    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6d380f-dafd-4263-b018-301261a1e223-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.330498 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.330504    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56dqt\" (UniqueName: \"kubernetes.io/projected/ab6d380f-dafd-4263-b018-301261a1e223-kube-api-access-56dqt\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:17:31.330774 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:31.330513    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6d380f-dafd-4263-b018-301261a1e223-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:17:32.026795 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:32.026750    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab6d380f-dafd-4263-b018-301261a1e223" containerID="4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b" exitCode=0
Jul 09 18:17:32.027279 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:32.026817    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"
Jul 09 18:17:32.027279 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:32.026841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj" event={"ID":"ab6d380f-dafd-4263-b018-301261a1e223","Type":"ContainerDied","Data":"4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b"}
Jul 09 18:17:32.027279 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:32.026899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj" event={"ID":"ab6d380f-dafd-4263-b018-301261a1e223","Type":"ContainerDied","Data":"cf8cf730e70abb7509d9d856945a0a2c27465d8f3ed2ad68215293d154dc0142"}
Jul 09 18:17:32.027279 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:32.026940    2565 scope.go:117] "RemoveContainer" containerID="4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b"
Jul 09 18:17:32.048707 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:32.048669    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"]
Jul 09 18:17:32.052363 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:32.052335    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj"]
Jul 09 18:17:32.869005 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:32.868970    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab6d380f-dafd-4263-b018-301261a1e223" path="/var/lib/kubelet/pods/ab6d380f-dafd-4263-b018-301261a1e223/volumes"
Jul 09 18:17:46.252196 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:46.251874    2565 scope.go:117] "RemoveContainer" containerID="4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b"
Jul 09 18:17:46.252506 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:17:46.252288    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b\": container with ID starting with 4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b not found: ID does not exist" containerID="4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b"
Jul 09 18:17:46.252506 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:46.252327    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b"} err="failed to get container status \"4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b\": rpc error: code = NotFound desc = could not find container \"4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b\": container with ID starting with 4d2948e259777d9d3c5f220f59f8ee5ca5a5c95131fa842df3c9d837c349142b not found: ID does not exist"
Jul 09 18:17:47.109830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:47.109744    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj_cd158fd4-7340-4858-b9d9-99c29c4fc6cb/main/0.log"
Jul 09 18:17:47.110362 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:47.110320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerStarted","Data":"ae67cd500758082782dff421264bb013b44518779b572abeb1469ba15b67abc3"}
Jul 09 18:17:47.110732 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:47.110705    2565 scope.go:117] "RemoveContainer" containerID="404901fd2c81070464238bfeec899ebc8d415a4c4ad463cf5878cf80b493b934"
Jul 09 18:17:48.118089 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:48.118062    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj_cd158fd4-7340-4858-b9d9-99c29c4fc6cb/main/0.log"
Jul 09 18:17:48.118576 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:48.118553    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerStarted","Data":"08ed4f46727c75d0e8f4ae926844042066670bed290c6b91375f6d471def3215"}
Jul 09 18:17:48.118751 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:48.118736    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:53.585841 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:53.585798    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:53.586449 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:53.585882    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:17:53.587072 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:53.587037    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.29:8082/healthz\": dial tcp 10.134.0.29:8082: connect: connection refused"
Jul 09 18:17:53.587176 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:17:53.587061    2565 logging.go:55] [core] [Channel #17 SubChannel #18]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.29:9003", ServerName: "10.134.0.29:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.29:9003: connect: connection refused"
Jul 09 18:17:54.586477 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:17:54.586426    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.29:9003\" within 1s: context deadline exceeded"
Jul 09 18:18:03.586366 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:18:03.586287    2565 logging.go:55] [core] [Channel #19 SubChannel #20]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.29:9003", ServerName: "10.134.0.29:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.29:9003: connect: connection refused"
Jul 09 18:18:03.587750 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:03.587728    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:18:03.588962 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:03.588943    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:18:03.612989 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:03.612938    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" podStartSLOduration=18.221666582 podStartE2EDuration="50.612920657s" totalImagesPullingTime="31.724364402s" totalInitContainerRuntime="666.889673ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:48.150022233 +0000 UTC m=+787.868284998" watchObservedRunningTime="2026-07-09 18:18:03.612920657 +0000 UTC m=+803.331183419"
Jul 09 18:18:04.586793 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:04.586750    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.29:9003\" within 1s: context deadline exceeded"
Jul 09 18:18:04.587262 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:18:04.586949    2565 logging.go:55] [core] [Channel #19 SubChannel #20]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.29:9003", ServerName: "10.134.0.29:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.29:9003: operation was canceled"
Jul 09 18:18:24.198218 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:24.198189    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:18:25.754760 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:25.754730    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"]
Jul 09 18:18:25.758375 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:25.758347    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"]
Jul 09 18:18:25.758642 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:25.758597    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="tokenizer" containerID="cri-o://ae67cd500758082782dff421264bb013b44518779b572abeb1469ba15b67abc3" gracePeriod=60
Jul 09 18:18:25.769124 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:18:25.769089    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:18:25.769124 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:18:25.769124 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:18:25.769124 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="08ed4f46727c75d0e8f4ae926844042066670bed290c6b91375f6d471def3215" cmd=["/bin/sleep","15"]
Jul 09 18:18:25.769379 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:25.769152    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" message="" err=<
Jul 09 18:18:25.769379 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:18:25.769379 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:18:25.769379 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:18:25.769379 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:18:25.769170    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:18:25.769379 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:18:25.769379 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:18:25.769379 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main" containerID="cri-o://08ed4f46727c75d0e8f4ae926844042066670bed290c6b91375f6d471def3215"
Jul 09 18:18:25.769379 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:25.769215    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main" containerID="cri-o://08ed4f46727c75d0e8f4ae926844042066670bed290c6b91375f6d471def3215" gracePeriod=60
Jul 09 18:18:26.298258 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:26.298229    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj_cd158fd4-7340-4858-b9d9-99c29c4fc6cb/main/0.log"
Jul 09 18:18:26.298630 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:26.298603    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerID="08ed4f46727c75d0e8f4ae926844042066670bed290c6b91375f6d471def3215" exitCode=0
Jul 09 18:18:26.298730 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:26.298679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerDied","Data":"08ed4f46727c75d0e8f4ae926844042066670bed290c6b91375f6d471def3215"}
Jul 09 18:18:26.298778 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:26.298769    2565 scope.go:117] "RemoveContainer" containerID="404901fd2c81070464238bfeec899ebc8d415a4c4ad463cf5878cf80b493b934"
Jul 09 18:18:27.305585 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.305555    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerID="ae67cd500758082782dff421264bb013b44518779b572abeb1469ba15b67abc3" exitCode=0
Jul 09 18:18:27.305893 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.305621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerDied","Data":"ae67cd500758082782dff421264bb013b44518779b572abeb1469ba15b67abc3"}
Jul 09 18:18:27.305893 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.305678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj" event={"ID":"cd158fd4-7340-4858-b9d9-99c29c4fc6cb","Type":"ContainerDied","Data":"24ed213fec9e226f0250667150bb52edc8d01a080e6e0697f92dae75b5eea34e"}
Jul 09 18:18:27.305893 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.305699    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ed213fec9e226f0250667150bb52edc8d01a080e6e0697f92dae75b5eea34e"
Jul 09 18:18:27.310011 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.309995    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:18:27.438950 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.438850    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-cache\") pod \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") "
Jul 09 18:18:27.438950 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.438893    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-uds\") pod \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") "
Jul 09 18:18:27.439145 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.438988    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dqcc\" (UniqueName: \"kubernetes.io/projected/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kube-api-access-9dqcc\") pod \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") "
Jul 09 18:18:27.439145 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.439030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tls-certs\") pod \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") "
Jul 09 18:18:27.439145 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.439070    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-tmp\") pod \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") "
Jul 09 18:18:27.439145 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.439141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kserve-provision-location\") pod \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\" (UID: \"cd158fd4-7340-4858-b9d9-99c29c4fc6cb\") "
Jul 09 18:18:27.439342 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.439156    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "cd158fd4-7340-4858-b9d9-99c29c4fc6cb" (UID: "cd158fd4-7340-4858-b9d9-99c29c4fc6cb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:27.439342 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.439254    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "cd158fd4-7340-4858-b9d9-99c29c4fc6cb" (UID: "cd158fd4-7340-4858-b9d9-99c29c4fc6cb"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:27.439453 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.439417    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:27.439453 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.439434    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:27.439553 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.439494    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "cd158fd4-7340-4858-b9d9-99c29c4fc6cb" (UID: "cd158fd4-7340-4858-b9d9-99c29c4fc6cb"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:27.439848 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.439828    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd158fd4-7340-4858-b9d9-99c29c4fc6cb" (UID: "cd158fd4-7340-4858-b9d9-99c29c4fc6cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:27.441200 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.441171    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd158fd4-7340-4858-b9d9-99c29c4fc6cb" (UID: "cd158fd4-7340-4858-b9d9-99c29c4fc6cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:27.441316 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.441222    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kube-api-access-9dqcc" (OuterVolumeSpecName: "kube-api-access-9dqcc") pod "cd158fd4-7340-4858-b9d9-99c29c4fc6cb" (UID: "cd158fd4-7340-4858-b9d9-99c29c4fc6cb"). InnerVolumeSpecName "kube-api-access-9dqcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:27.540360 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.540324    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dqcc\" (UniqueName: \"kubernetes.io/projected/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kube-api-access-9dqcc\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:27.540360 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.540355    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:27.540360 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.540367    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-tokenizer-tmp\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:27.540585 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:27.540375    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd158fd4-7340-4858-b9d9-99c29c4fc6cb-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:28.309696 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:28.309665    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"
Jul 09 18:18:28.330490 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:28.330459    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"]
Jul 09 18:18:28.334626 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:28.334603    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj"]
Jul 09 18:18:28.867699 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:28.867664    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" path="/var/lib/kubelet/pods/cd158fd4-7340-4858-b9d9-99c29c4fc6cb/volumes"
Jul 09 18:18:40.779182 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:40.779137    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z" podUID="b313bdf0-269c-4a97-b216-ded35085a7f5" containerName="main" containerID="cri-o://21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8" gracePeriod=45
Jul 09 18:18:41.046209 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.046148    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:18:41.165726 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.165695    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-dshm\") pod \"b313bdf0-269c-4a97-b216-ded35085a7f5\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") "
Jul 09 18:18:41.165726 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.165729    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-tmp-dir\") pod \"b313bdf0-269c-4a97-b216-ded35085a7f5\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") "
Jul 09 18:18:41.165966 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.165786    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-kserve-provision-location\") pod \"b313bdf0-269c-4a97-b216-ded35085a7f5\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") "
Jul 09 18:18:41.165966 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.165839    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-model-cache\") pod \"b313bdf0-269c-4a97-b216-ded35085a7f5\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") "
Jul 09 18:18:41.165966 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.165875    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-home\") pod \"b313bdf0-269c-4a97-b216-ded35085a7f5\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") "
Jul 09 18:18:41.165966 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.165965    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b313bdf0-269c-4a97-b216-ded35085a7f5-tls-certs\") pod \"b313bdf0-269c-4a97-b216-ded35085a7f5\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") "
Jul 09 18:18:41.166173 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.165990    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm4lc\" (UniqueName: \"kubernetes.io/projected/b313bdf0-269c-4a97-b216-ded35085a7f5-kube-api-access-cm4lc\") pod \"b313bdf0-269c-4a97-b216-ded35085a7f5\" (UID: \"b313bdf0-269c-4a97-b216-ded35085a7f5\") "
Jul 09 18:18:41.166230 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.166142    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b313bdf0-269c-4a97-b216-ded35085a7f5" (UID: "b313bdf0-269c-4a97-b216-ded35085a7f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:41.166230 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.166206    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b313bdf0-269c-4a97-b216-ded35085a7f5" (UID: "b313bdf0-269c-4a97-b216-ded35085a7f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:41.166310 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.166270    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:41.166467 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.166439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-home" (OuterVolumeSpecName: "home") pod "b313bdf0-269c-4a97-b216-ded35085a7f5" (UID: "b313bdf0-269c-4a97-b216-ded35085a7f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:41.167917 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.167872    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-dshm" (OuterVolumeSpecName: "dshm") pod "b313bdf0-269c-4a97-b216-ded35085a7f5" (UID: "b313bdf0-269c-4a97-b216-ded35085a7f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:41.168587 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.168561    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b313bdf0-269c-4a97-b216-ded35085a7f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b313bdf0-269c-4a97-b216-ded35085a7f5" (UID: "b313bdf0-269c-4a97-b216-ded35085a7f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:41.168667 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.168587    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b313bdf0-269c-4a97-b216-ded35085a7f5-kube-api-access-cm4lc" (OuterVolumeSpecName: "kube-api-access-cm4lc") pod "b313bdf0-269c-4a97-b216-ded35085a7f5" (UID: "b313bdf0-269c-4a97-b216-ded35085a7f5"). InnerVolumeSpecName "kube-api-access-cm4lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:41.220262 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.220222    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b313bdf0-269c-4a97-b216-ded35085a7f5" (UID: "b313bdf0-269c-4a97-b216-ded35085a7f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:41.267716 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.267680    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:41.267716 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.267711    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b313bdf0-269c-4a97-b216-ded35085a7f5-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:41.267716 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.267724    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm4lc\" (UniqueName: \"kubernetes.io/projected/b313bdf0-269c-4a97-b216-ded35085a7f5-kube-api-access-cm4lc\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:41.267938 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.267733    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:41.267938 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.267742    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:41.267938 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.267750    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b313bdf0-269c-4a97-b216-ded35085a7f5-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:18:41.366450 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.366354    2565 generic.go:356] "Generic (PLEG): container finished" podID="b313bdf0-269c-4a97-b216-ded35085a7f5" containerID="21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8" exitCode=0
Jul 09 18:18:41.366450 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.366404    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"
Jul 09 18:18:41.366450 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.366421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z" event={"ID":"b313bdf0-269c-4a97-b216-ded35085a7f5","Type":"ContainerDied","Data":"21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8"}
Jul 09 18:18:41.366655 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.366494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z" event={"ID":"b313bdf0-269c-4a97-b216-ded35085a7f5","Type":"ContainerDied","Data":"6d8ebb82f527dcbc865bec7a6211292ef4b18704d7d53f9cc8065574f6da67ab"}
Jul 09 18:18:41.366655 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.366518    2565 scope.go:117] "RemoveContainer" containerID="21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8"
Jul 09 18:18:41.375681 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.375662    2565 scope.go:117] "RemoveContainer" containerID="f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7"
Jul 09 18:18:41.383800 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.383782    2565 scope.go:117] "RemoveContainer" containerID="21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8"
Jul 09 18:18:41.384059 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:18:41.384042    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8\": container with ID starting with 21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8 not found: ID does not exist" containerID="21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8"
Jul 09 18:18:41.384114 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.384068    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8"} err="failed to get container status \"21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8\": rpc error: code = NotFound desc = could not find container \"21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8\": container with ID starting with 21c0264ce84461fb0ee5c2b014f52ebd04908b8fcbe98747a944572dd37609e8 not found: ID does not exist"
Jul 09 18:18:41.384114 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.384085    2565 scope.go:117] "RemoveContainer" containerID="f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7"
Jul 09 18:18:41.384336 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:18:41.384318    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7\": container with ID starting with f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7 not found: ID does not exist" containerID="f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7"
Jul 09 18:18:41.384383 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.384342    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7"} err="failed to get container status \"f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7\": rpc error: code = NotFound desc = could not find container \"f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7\": container with ID starting with f7b871bbaa556d30f79530eb18476a44a208017b6cbe9e6a02683798a6d2c5e7 not found: ID does not exist"
Jul 09 18:18:41.388990 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.388972    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"]
Jul 09 18:18:41.394963 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.394940    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z"]
Jul 09 18:18:41.977063 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:41.977023    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z" podUID="b313bdf0-269c-4a97-b216-ded35085a7f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:18:42.867701 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:42.867670    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b313bdf0-269c-4a97-b216-ded35085a7f5" path="/var/lib/kubelet/pods/b313bdf0-269c-4a97-b216-ded35085a7f5/volumes"
Jul 09 18:18:57.805837 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.805789    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"]
Jul 09 18:18:57.806520 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806499    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="storage-initializer"
Jul 09 18:18:57.806598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806524    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="storage-initializer"
Jul 09 18:18:57.806598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806535    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main"
Jul 09 18:18:57.806598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806544    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main"
Jul 09 18:18:57.806598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806560    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b313bdf0-269c-4a97-b216-ded35085a7f5" containerName="storage-initializer"
Jul 09 18:18:57.806598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806570    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b313bdf0-269c-4a97-b216-ded35085a7f5" containerName="storage-initializer"
Jul 09 18:18:57.806598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806587    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b313bdf0-269c-4a97-b216-ded35085a7f5" containerName="main"
Jul 09 18:18:57.806598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806596    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b313bdf0-269c-4a97-b216-ded35085a7f5" containerName="main"
Jul 09 18:18:57.806934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806643    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab6d380f-dafd-4263-b018-301261a1e223" containerName="main"
Jul 09 18:18:57.806934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806657    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab6d380f-dafd-4263-b018-301261a1e223" containerName="main"
Jul 09 18:18:57.806934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806668    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="tokenizer"
Jul 09 18:18:57.806934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806677    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="tokenizer"
Jul 09 18:18:57.806934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806828    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main"
Jul 09 18:18:57.806934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806844    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="tokenizer"
Jul 09 18:18:57.806934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806855    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab6d380f-dafd-4263-b018-301261a1e223" containerName="main"
Jul 09 18:18:57.806934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806862    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main"
Jul 09 18:18:57.806934 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.806881    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b313bdf0-269c-4a97-b216-ded35085a7f5" containerName="main"
Jul 09 18:18:57.811002 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.810972    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:57.814425 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.814403    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:18:57.814536 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.814403    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s8z92\""
Jul 09 18:18:57.815315 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.815301    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 09 18:18:57.815370 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.815321    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:18:57.818950 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.818924    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"]
Jul 09 18:18:57.909545 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.909503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:57.909724 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.909562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:57.909724 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.909593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f240a74a-0591-42cd-a874-11d53fe253a2-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:57.909724 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.909642    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:57.909724 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.909666    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:57.909724 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:57.909695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqgsb\" (UniqueName: \"kubernetes.io/projected/f240a74a-0591-42cd-a874-11d53fe253a2-kube-api-access-nqgsb\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.010875 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.010839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.010875 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.010873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f240a74a-0591-42cd-a874-11d53fe253a2-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.011122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.010940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.011122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.010960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.011122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.010976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqgsb\" (UniqueName: \"kubernetes.io/projected/f240a74a-0591-42cd-a874-11d53fe253a2-kube-api-access-nqgsb\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.011122 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.011019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.011326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.011303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.011326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.011320    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.011422 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.011396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.013242 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.013223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.013392 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.013375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f240a74a-0591-42cd-a874-11d53fe253a2-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.021099 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.021074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqgsb\" (UniqueName: \"kubernetes.io/projected/f240a74a-0591-42cd-a874-11d53fe253a2-kube-api-access-nqgsb\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.121588 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.121500    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:58.197494 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.197456    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"]
Jul 09 18:18:58.198034 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.198011    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main"
Jul 09 18:18:58.198034 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.198034    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd158fd4-7340-4858-b9d9-99c29c4fc6cb" containerName="main"
Jul 09 18:18:58.200474 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.200456    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.203495 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.203474    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-7rv6v\""
Jul 09 18:18:58.212271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.212234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.212503 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.212483    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwsj\" (UniqueName: \"kubernetes.io/projected/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-kube-api-access-wzwsj\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.212716 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.212700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.213281 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.213260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"]
Jul 09 18:18:58.265957 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.265930    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"]
Jul 09 18:18:58.269759 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:18:58.269728    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf240a74a_0591_42cd_a874_11d53fe253a2.slice/crio-43e0734ca68cfd3982f187b54cbfc3221f0d1ae5526a46ab01eee69ae4f7fae3 WatchSource:0}: Error finding container 43e0734ca68cfd3982f187b54cbfc3221f0d1ae5526a46ab01eee69ae4f7fae3: Status 404 returned error can't find the container with id 43e0734ca68cfd3982f187b54cbfc3221f0d1ae5526a46ab01eee69ae4f7fae3
Jul 09 18:18:58.313228 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.313202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwsj\" (UniqueName: \"kubernetes.io/projected/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-kube-api-access-wzwsj\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.313338 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.313268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.313338 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.313332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.313642 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.313621    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.315784 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.315762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.322532 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.322512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwsj\" (UniqueName: \"kubernetes.io/projected/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-kube-api-access-wzwsj\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.444010 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.443977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs" event={"ID":"f240a74a-0591-42cd-a874-11d53fe253a2","Type":"ContainerStarted","Data":"0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6"}
Jul 09 18:18:58.444168 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.444032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs" event={"ID":"f240a74a-0591-42cd-a874-11d53fe253a2","Type":"ContainerStarted","Data":"43e0734ca68cfd3982f187b54cbfc3221f0d1ae5526a46ab01eee69ae4f7fae3"}
Jul 09 18:18:58.466836 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.466782    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs" podStartSLOduration=1.466765223 podStartE2EDuration="1.466765223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:58.465113985 +0000 UTC m=+858.183376749" watchObservedRunningTime="2026-07-09 18:18:58.466765223 +0000 UTC m=+858.185027986"
Jul 09 18:18:58.511121 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.511081    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:58.644118 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:58.644094    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"]
Jul 09 18:18:58.646061 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:18:58.646038    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec8d1dc_4482_4154_bd3c_aded9b4b44eb.slice/crio-9bbdbdc595e4e58f958a1f2f4d97c0d0a424c0110bde6e60756cc50937ee14f9 WatchSource:0}: Error finding container 9bbdbdc595e4e58f958a1f2f4d97c0d0a424c0110bde6e60756cc50937ee14f9: Status 404 returned error can't find the container with id 9bbdbdc595e4e58f958a1f2f4d97c0d0a424c0110bde6e60756cc50937ee14f9
Jul 09 18:18:59.122412 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:59.122367    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:18:59.449529 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:59.449488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6" event={"ID":"fec8d1dc-4482-4154-bd3c-aded9b4b44eb","Type":"ContainerStarted","Data":"bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74"}
Jul 09 18:18:59.449720 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:59.449549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6" event={"ID":"fec8d1dc-4482-4154-bd3c-aded9b4b44eb","Type":"ContainerStarted","Data":"9bbdbdc595e4e58f958a1f2f4d97c0d0a424c0110bde6e60756cc50937ee14f9"}
Jul 09 18:18:59.449720 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:59.449664    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:18:59.467280 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:18:59.467227    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6" podStartSLOduration=1.467208049 podStartE2EDuration="1.467208049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:59.466938953 +0000 UTC m=+859.185201714" watchObservedRunningTime="2026-07-09 18:18:59.467208049 +0000 UTC m=+859.185470814"
Jul 09 18:19:08.122556 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:08.122518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:19:08.134918 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:08.134873    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:19:08.507814 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:08.507783    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:19:30.455983 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:30.455943    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:19:31.634032 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.633933    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"]
Jul 09 18:19:31.636116 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.636089    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"]
Jul 09 18:19:31.662274 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:19:31.662236    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:19:31.662274 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:19:31.662274 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:19:31.662274 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74" cmd=["/bin/sleep","15"]
Jul 09 18:19:31.662531 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.662296    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6" message="" err=<
Jul 09 18:19:31.662531 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:19:31.662531 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:19:31.662531 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:19:31.662531 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:19:31.662313    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:19:31.662531 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:19:31.662531 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:19:31.662531 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6" podUID="fec8d1dc-4482-4154-bd3c-aded9b4b44eb" containerName="main" containerID="cri-o://bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74"
Jul 09 18:19:31.662531 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.662359    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6" podUID="fec8d1dc-4482-4154-bd3c-aded9b4b44eb" containerName="main" containerID="cri-o://bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74" gracePeriod=60
Jul 09 18:19:31.909602 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.909576    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:19:31.993031 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.992992    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tokenizer-uds\") pod \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") "
Jul 09 18:19:31.993225 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.993077    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tls-certs\") pod \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") "
Jul 09 18:19:31.993225 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.993120    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzwsj\" (UniqueName: \"kubernetes.io/projected/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-kube-api-access-wzwsj\") pod \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\" (UID: \"fec8d1dc-4482-4154-bd3c-aded9b4b44eb\") "
Jul 09 18:19:31.993347 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.993316    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "fec8d1dc-4482-4154-bd3c-aded9b4b44eb" (UID: "fec8d1dc-4482-4154-bd3c-aded9b4b44eb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:31.995318 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.995287    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fec8d1dc-4482-4154-bd3c-aded9b4b44eb" (UID: "fec8d1dc-4482-4154-bd3c-aded9b4b44eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:19:31.995318 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:31.995293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-kube-api-access-wzwsj" (OuterVolumeSpecName: "kube-api-access-wzwsj") pod "fec8d1dc-4482-4154-bd3c-aded9b4b44eb" (UID: "fec8d1dc-4482-4154-bd3c-aded9b4b44eb"). InnerVolumeSpecName "kube-api-access-wzwsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:19:32.094090 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.094041    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzwsj\" (UniqueName: \"kubernetes.io/projected/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-kube-api-access-wzwsj\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:19:32.094090 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.094085    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:19:32.094090 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.094097    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec8d1dc-4482-4154-bd3c-aded9b4b44eb-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:19:32.604325 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.604279    2565 generic.go:356] "Generic (PLEG): container finished" podID="fec8d1dc-4482-4154-bd3c-aded9b4b44eb" containerID="bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74" exitCode=0
Jul 09 18:19:32.604325 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.604325    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"
Jul 09 18:19:32.604595 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.604349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6" event={"ID":"fec8d1dc-4482-4154-bd3c-aded9b4b44eb","Type":"ContainerDied","Data":"bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74"}
Jul 09 18:19:32.604595 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.604403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6" event={"ID":"fec8d1dc-4482-4154-bd3c-aded9b4b44eb","Type":"ContainerDied","Data":"9bbdbdc595e4e58f958a1f2f4d97c0d0a424c0110bde6e60756cc50937ee14f9"}
Jul 09 18:19:32.604595 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.604429    2565 scope.go:117] "RemoveContainer" containerID="bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74"
Jul 09 18:19:32.615377 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.615357    2565 scope.go:117] "RemoveContainer" containerID="bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74"
Jul 09 18:19:32.615670 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:19:32.615648    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74\": container with ID starting with bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74 not found: ID does not exist" containerID="bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74"
Jul 09 18:19:32.615771 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.615680    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74"} err="failed to get container status \"bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74\": rpc error: code = NotFound desc = could not find container \"bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74\": container with ID starting with bfbb0544bfb3d4f58615d33546e6577555452006b7f6f520768de3f9880fdb74 not found: ID does not exist"
Jul 09 18:19:32.627120 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.627097    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"]
Jul 09 18:19:32.634799 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.634632    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6"]
Jul 09 18:19:32.867868 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:32.867787    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec8d1dc-4482-4154-bd3c-aded9b4b44eb" path="/var/lib/kubelet/pods/fec8d1dc-4482-4154-bd3c-aded9b4b44eb/volumes"
Jul 09 18:19:40.728449 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:40.728422    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:19:40.728878 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:40.728422    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:19:46.675343 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:46.675230    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs" podUID="f240a74a-0591-42cd-a874-11d53fe253a2" containerName="main" containerID="cri-o://0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6" gracePeriod=45
Jul 09 18:19:46.932766 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:46.932707    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:19:47.127165 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.127131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-tmp-dir\") pod \"f240a74a-0591-42cd-a874-11d53fe253a2\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") "
Jul 09 18:19:47.127327 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.127191    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqgsb\" (UniqueName: \"kubernetes.io/projected/f240a74a-0591-42cd-a874-11d53fe253a2-kube-api-access-nqgsb\") pod \"f240a74a-0591-42cd-a874-11d53fe253a2\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") "
Jul 09 18:19:47.127327 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.127221    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-model-cache\") pod \"f240a74a-0591-42cd-a874-11d53fe253a2\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") "
Jul 09 18:19:47.127327 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.127257    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f240a74a-0591-42cd-a874-11d53fe253a2-tls-certs\") pod \"f240a74a-0591-42cd-a874-11d53fe253a2\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") "
Jul 09 18:19:47.127327 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.127275    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-home\") pod \"f240a74a-0591-42cd-a874-11d53fe253a2\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") "
Jul 09 18:19:47.127327 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.127295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-dshm\") pod \"f240a74a-0591-42cd-a874-11d53fe253a2\" (UID: \"f240a74a-0591-42cd-a874-11d53fe253a2\") "
Jul 09 18:19:47.127564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.127453    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f240a74a-0591-42cd-a874-11d53fe253a2" (UID: "f240a74a-0591-42cd-a874-11d53fe253a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:47.127564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.127539    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f240a74a-0591-42cd-a874-11d53fe253a2" (UID: "f240a74a-0591-42cd-a874-11d53fe253a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:47.127673 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.127552    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-home" (OuterVolumeSpecName: "home") pod "f240a74a-0591-42cd-a874-11d53fe253a2" (UID: "f240a74a-0591-42cd-a874-11d53fe253a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:47.129398 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.129375    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f240a74a-0591-42cd-a874-11d53fe253a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f240a74a-0591-42cd-a874-11d53fe253a2" (UID: "f240a74a-0591-42cd-a874-11d53fe253a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:19:47.129502 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.129447    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f240a74a-0591-42cd-a874-11d53fe253a2-kube-api-access-nqgsb" (OuterVolumeSpecName: "kube-api-access-nqgsb") pod "f240a74a-0591-42cd-a874-11d53fe253a2" (UID: "f240a74a-0591-42cd-a874-11d53fe253a2"). InnerVolumeSpecName "kube-api-access-nqgsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:19:47.129502 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.129466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-dshm" (OuterVolumeSpecName: "dshm") pod "f240a74a-0591-42cd-a874-11d53fe253a2" (UID: "f240a74a-0591-42cd-a874-11d53fe253a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:47.227968 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.227934    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.227968 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.227966    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqgsb\" (UniqueName: \"kubernetes.io/projected/f240a74a-0591-42cd-a874-11d53fe253a2-kube-api-access-nqgsb\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.228176 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.227977    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.228176 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.227986    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f240a74a-0591-42cd-a874-11d53fe253a2-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.228176 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.227994    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.228176 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.228000    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f240a74a-0591-42cd-a874-11d53fe253a2-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.675444 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.675353    2565 generic.go:356] "Generic (PLEG): container finished" podID="f240a74a-0591-42cd-a874-11d53fe253a2" containerID="0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6" exitCode=0
Jul 09 18:19:47.675444 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.675400    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"
Jul 09 18:19:47.675444 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.675425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs" event={"ID":"f240a74a-0591-42cd-a874-11d53fe253a2","Type":"ContainerDied","Data":"0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6"}
Jul 09 18:19:47.675993 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.675474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs" event={"ID":"f240a74a-0591-42cd-a874-11d53fe253a2","Type":"ContainerDied","Data":"43e0734ca68cfd3982f187b54cbfc3221f0d1ae5526a46ab01eee69ae4f7fae3"}
Jul 09 18:19:47.675993 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.675502    2565 scope.go:117] "RemoveContainer" containerID="0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6"
Jul 09 18:19:47.684414 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.684395    2565 scope.go:117] "RemoveContainer" containerID="0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6"
Jul 09 18:19:47.684716 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:19:47.684696    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6\": container with ID starting with 0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6 not found: ID does not exist" containerID="0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6"
Jul 09 18:19:47.684786 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.684729    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6"} err="failed to get container status \"0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6\": rpc error: code = NotFound desc = could not find container \"0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6\": container with ID starting with 0ec2b181a0d11518d791d598aa9a59bb0f92400d7050faba4eeb96fd6ea25dd6 not found: ID does not exist"
Jul 09 18:19:47.698666 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.698633    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"]
Jul 09 18:19:47.702335 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:47.702307    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs"]
Jul 09 18:19:48.867424 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:19:48.867391    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f240a74a-0591-42cd-a874-11d53fe253a2" path="/var/lib/kubelet/pods/f240a74a-0591-42cd-a874-11d53fe253a2/volumes"
Jul 09 18:20:21.671607 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.671575    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"]
Jul 09 18:20:21.672138 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.671923    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f240a74a-0591-42cd-a874-11d53fe253a2" containerName="main"
Jul 09 18:20:21.672138 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.671939    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f240a74a-0591-42cd-a874-11d53fe253a2" containerName="main"
Jul 09 18:20:21.672138 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.671961    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec8d1dc-4482-4154-bd3c-aded9b4b44eb" containerName="main"
Jul 09 18:20:21.672138 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.671966    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec8d1dc-4482-4154-bd3c-aded9b4b44eb" containerName="main"
Jul 09 18:20:21.672138 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.672016    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f240a74a-0591-42cd-a874-11d53fe253a2" containerName="main"
Jul 09 18:20:21.672138 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.672024    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec8d1dc-4482-4154-bd3c-aded9b4b44eb" containerName="main"
Jul 09 18:20:21.673953 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.673932    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.676975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.676951    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:20:21.677114 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.677093    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 09 18:20:21.677168 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.677115    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s8z92\""
Jul 09 18:20:21.677168 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.677145    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:20:21.688070 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.688045    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"]
Jul 09 18:20:21.803669 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.803632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.803866 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.803672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.803866 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.803771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.803866 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.803810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnlhs\" (UniqueName: \"kubernetes.io/projected/e13dd592-502b-4d60-adcd-0caa4b5fabdc-kube-api-access-tnlhs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.803866 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.803838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.803866 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.803861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.893974 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.893938    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"]
Jul 09 18:20:21.900544 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.900520    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:21.903299 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.903251    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-v6xc2\""
Jul 09 18:20:21.904633 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.904609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.904739 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.904643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.904739 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.904679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.904856 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.904818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnlhs\" (UniqueName: \"kubernetes.io/projected/e13dd592-502b-4d60-adcd-0caa4b5fabdc-kube-api-access-tnlhs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.904947 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.904865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.904947 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.904895    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.905060 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.905032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.905238 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.905216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.905342 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.905269    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.907580 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.907552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.908006 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.907985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.909749 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.909729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"]
Jul 09 18:20:21.913751 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.913726    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnlhs\" (UniqueName: \"kubernetes.io/projected/e13dd592-502b-4d60-adcd-0caa4b5fabdc-kube-api-access-tnlhs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:21.984802 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:21.984763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:22.005931 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.005884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b167043-900e-43a5-ab25-f6fdc9b1e683-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.006077 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.005975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mgv\" (UniqueName: \"kubernetes.io/projected/9b167043-900e-43a5-ab25-f6fdc9b1e683-kube-api-access-t5mgv\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.006077 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.006027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b167043-900e-43a5-ab25-f6fdc9b1e683-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.107053 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.107012    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b167043-900e-43a5-ab25-f6fdc9b1e683-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.107227 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.107099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b167043-900e-43a5-ab25-f6fdc9b1e683-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.107227 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.107134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mgv\" (UniqueName: \"kubernetes.io/projected/9b167043-900e-43a5-ab25-f6fdc9b1e683-kube-api-access-t5mgv\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.107583 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.107557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b167043-900e-43a5-ab25-f6fdc9b1e683-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.109364 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.109344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b167043-900e-43a5-ab25-f6fdc9b1e683-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.114986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.114967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mgv\" (UniqueName: \"kubernetes.io/projected/9b167043-900e-43a5-ab25-f6fdc9b1e683-kube-api-access-t5mgv\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.119603 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.119581    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"]
Jul 09 18:20:22.122931 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:20:22.122893    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13dd592_502b_4d60_adcd_0caa4b5fabdc.slice/crio-2457afab5f78695308e2e4105cb759399d8d74e360117718558f77beb66308e6 WatchSource:0}: Error finding container 2457afab5f78695308e2e4105cb759399d8d74e360117718558f77beb66308e6: Status 404 returned error can't find the container with id 2457afab5f78695308e2e4105cb759399d8d74e360117718558f77beb66308e6
Jul 09 18:20:22.225304 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.225270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.355085 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.355052    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"]
Jul 09 18:20:22.358639 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:20:22.358605    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b167043_900e_43a5_ab25_f6fdc9b1e683.slice/crio-cccd337e4080245c58edb2638a86fbd33f3911186ff4aa28911eb68cda086ae8 WatchSource:0}: Error finding container cccd337e4080245c58edb2638a86fbd33f3911186ff4aa28911eb68cda086ae8: Status 404 returned error can't find the container with id cccd337e4080245c58edb2638a86fbd33f3911186ff4aa28911eb68cda086ae8
Jul 09 18:20:22.843770 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.843727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6" event={"ID":"9b167043-900e-43a5-ab25-f6fdc9b1e683","Type":"ContainerStarted","Data":"659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c"}
Jul 09 18:20:22.844242 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.843784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6" event={"ID":"9b167043-900e-43a5-ab25-f6fdc9b1e683","Type":"ContainerStarted","Data":"cccd337e4080245c58edb2638a86fbd33f3911186ff4aa28911eb68cda086ae8"}
Jul 09 18:20:22.844242 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.843832    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:20:22.845326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.845303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk" event={"ID":"e13dd592-502b-4d60-adcd-0caa4b5fabdc","Type":"ContainerStarted","Data":"54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756"}
Jul 09 18:20:22.845428 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.845342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk" event={"ID":"e13dd592-502b-4d60-adcd-0caa4b5fabdc","Type":"ContainerStarted","Data":"2457afab5f78695308e2e4105cb759399d8d74e360117718558f77beb66308e6"}
Jul 09 18:20:22.861971 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.861927    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6" podStartSLOduration=1.861895318 podStartE2EDuration="1.861895318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:20:22.860521597 +0000 UTC m=+942.578784362" watchObservedRunningTime="2026-07-09 18:20:22.861895318 +0000 UTC m=+942.580158082"
Jul 09 18:20:22.882663 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.882617    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk" podStartSLOduration=1.882601789 podStartE2EDuration="1.882601789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:20:22.880955684 +0000 UTC m=+942.599218441" watchObservedRunningTime="2026-07-09 18:20:22.882601789 +0000 UTC m=+942.600864552"
Jul 09 18:20:22.985870 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:22.985821    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:31.985977 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:31.985926    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:31.998922 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:31.998881    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:32.899732 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:32.899699    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:20:53.852801 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:20:53.852771    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:21:48.520230 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.520188    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"]
Jul 09 18:21:48.523436 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.523419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.527249 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.527218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-5hnc8\""
Jul 09 18:21:48.527388 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.527230    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 09 18:21:48.527388 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.527243    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:21:48.534048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.534017    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"]
Jul 09 18:21:48.648920 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.648874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.649110 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.649014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.649110 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.649049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9zn\" (UniqueName: \"kubernetes.io/projected/6166416a-f536-4de9-bf76-82f8648d9bf0-kube-api-access-2s9zn\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.649225 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.649136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6166416a-f536-4de9-bf76-82f8648d9bf0-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.749740 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.749699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.749988 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.749774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.749988 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.749807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9zn\" (UniqueName: \"kubernetes.io/projected/6166416a-f536-4de9-bf76-82f8648d9bf0-kube-api-access-2s9zn\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.749988 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.749849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6166416a-f536-4de9-bf76-82f8648d9bf0-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.750212 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.750190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.750272 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.750226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.752240 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.752222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6166416a-f536-4de9-bf76-82f8648d9bf0-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.758514 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.758487    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9zn\" (UniqueName: \"kubernetes.io/projected/6166416a-f536-4de9-bf76-82f8648d9bf0-kube-api-access-2s9zn\") pod \"auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.833150 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.833059    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:48.974089 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:48.974064    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"]
Jul 09 18:21:48.976772 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:21:48.976749    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6166416a_f536_4de9_bf76_82f8648d9bf0.slice/crio-f95f0690a07f5670dea06c856075f153401558b77f0cdad1e5d0063e39149e51 WatchSource:0}: Error finding container f95f0690a07f5670dea06c856075f153401558b77f0cdad1e5d0063e39149e51: Status 404 returned error can't find the container with id f95f0690a07f5670dea06c856075f153401558b77f0cdad1e5d0063e39149e51
Jul 09 18:21:49.230706 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:49.230665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" event={"ID":"6166416a-f536-4de9-bf76-82f8648d9bf0","Type":"ContainerStarted","Data":"65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959"}
Jul 09 18:21:49.230882 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:49.230737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" event={"ID":"6166416a-f536-4de9-bf76-82f8648d9bf0","Type":"ContainerStarted","Data":"f95f0690a07f5670dea06c856075f153401558b77f0cdad1e5d0063e39149e51"}
Jul 09 18:21:50.236884 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:50.236849    2565 generic.go:356] "Generic (PLEG): container finished" podID="6166416a-f536-4de9-bf76-82f8648d9bf0" containerID="65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959" exitCode=0
Jul 09 18:21:50.237298 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:50.236948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" event={"ID":"6166416a-f536-4de9-bf76-82f8648d9bf0","Type":"ContainerDied","Data":"65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959"}
Jul 09 18:21:51.243236 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:51.243197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" event={"ID":"6166416a-f536-4de9-bf76-82f8648d9bf0","Type":"ContainerStarted","Data":"8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744"}
Jul 09 18:21:51.243621 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:51.243305    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:21:51.262792 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:21:51.262739    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" podStartSLOduration=2.599862554 podStartE2EDuration="3.262724987s" totalImagesPullingTime="0s" totalInitContainerRuntime="662.862433ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:51.261433156 +0000 UTC m=+1030.979695921" watchObservedRunningTime="2026-07-09 18:21:51.262724987 +0000 UTC m=+1030.980987751"
Jul 09 18:22:22.250148 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:22.250117    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:22:36.885438 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:36.885363    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"]
Jul 09 18:22:36.888360 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:36.888333    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"]
Jul 09 18:22:36.894992 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:22:36.894955    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:22:36.894992 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:22:36.894992 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:22:36.894992 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c" cmd=["/bin/sleep","15"]
Jul 09 18:22:36.895281 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:36.895007    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6" message="" err=<
Jul 09 18:22:36.895281 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:22:36.895281 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:22:36.895281 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:22:36.895281 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:22:36.895019    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:22:36.895281 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:22:36.895281 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:22:36.895281 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6" podUID="9b167043-900e-43a5-ab25-f6fdc9b1e683" containerName="main" containerID="cri-o://659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c"
Jul 09 18:22:36.895281 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:36.895065    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6" podUID="9b167043-900e-43a5-ab25-f6fdc9b1e683" containerName="main" containerID="cri-o://659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c" gracePeriod=60
Jul 09 18:22:37.151933 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.151894    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:22:37.268091 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.268057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5mgv\" (UniqueName: \"kubernetes.io/projected/9b167043-900e-43a5-ab25-f6fdc9b1e683-kube-api-access-t5mgv\") pod \"9b167043-900e-43a5-ab25-f6fdc9b1e683\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") "
Jul 09 18:22:37.268264 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.268145    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b167043-900e-43a5-ab25-f6fdc9b1e683-tls-certs\") pod \"9b167043-900e-43a5-ab25-f6fdc9b1e683\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") "
Jul 09 18:22:37.268264 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.268187    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b167043-900e-43a5-ab25-f6fdc9b1e683-tokenizer-uds\") pod \"9b167043-900e-43a5-ab25-f6fdc9b1e683\" (UID: \"9b167043-900e-43a5-ab25-f6fdc9b1e683\") "
Jul 09 18:22:37.268502 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.268476    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b167043-900e-43a5-ab25-f6fdc9b1e683-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9b167043-900e-43a5-ab25-f6fdc9b1e683" (UID: "9b167043-900e-43a5-ab25-f6fdc9b1e683"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:37.270193 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.270162    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b167043-900e-43a5-ab25-f6fdc9b1e683-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b167043-900e-43a5-ab25-f6fdc9b1e683" (UID: "9b167043-900e-43a5-ab25-f6fdc9b1e683"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:22:37.270282 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.270188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b167043-900e-43a5-ab25-f6fdc9b1e683-kube-api-access-t5mgv" (OuterVolumeSpecName: "kube-api-access-t5mgv") pod "9b167043-900e-43a5-ab25-f6fdc9b1e683" (UID: "9b167043-900e-43a5-ab25-f6fdc9b1e683"). InnerVolumeSpecName "kube-api-access-t5mgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:22:37.369692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.369647    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b167043-900e-43a5-ab25-f6fdc9b1e683-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:22:37.369692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.369686    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b167043-900e-43a5-ab25-f6fdc9b1e683-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:22:37.369692 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.369699    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mgv\" (UniqueName: \"kubernetes.io/projected/9b167043-900e-43a5-ab25-f6fdc9b1e683-kube-api-access-t5mgv\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:22:37.450681 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.450641    2565 generic.go:356] "Generic (PLEG): container finished" podID="9b167043-900e-43a5-ab25-f6fdc9b1e683" containerID="659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c" exitCode=0
Jul 09 18:22:37.450837 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.450684    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"
Jul 09 18:22:37.450837 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.450725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6" event={"ID":"9b167043-900e-43a5-ab25-f6fdc9b1e683","Type":"ContainerDied","Data":"659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c"}
Jul 09 18:22:37.450837 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.450778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6" event={"ID":"9b167043-900e-43a5-ab25-f6fdc9b1e683","Type":"ContainerDied","Data":"cccd337e4080245c58edb2638a86fbd33f3911186ff4aa28911eb68cda086ae8"}
Jul 09 18:22:37.450837 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.450805    2565 scope.go:117] "RemoveContainer" containerID="659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c"
Jul 09 18:22:37.459843 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.459826    2565 scope.go:117] "RemoveContainer" containerID="659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c"
Jul 09 18:22:37.460205 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:22:37.460184    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c\": container with ID starting with 659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c not found: ID does not exist" containerID="659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c"
Jul 09 18:22:37.460278 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.460213    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c"} err="failed to get container status \"659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c\": rpc error: code = NotFound desc = could not find container \"659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c\": container with ID starting with 659a460c559a30cdbaccfc7b0440cd7c835c99fd87442f46bb81aecdaa5cc47c not found: ID does not exist"
Jul 09 18:22:37.481485 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.481453    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"]
Jul 09 18:22:37.487849 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:37.487825    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6"]
Jul 09 18:22:38.868123 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:38.868091    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b167043-900e-43a5-ab25-f6fdc9b1e683" path="/var/lib/kubelet/pods/9b167043-900e-43a5-ab25-f6fdc9b1e683/volumes"
Jul 09 18:22:51.911127 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:51.911060    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk" podUID="e13dd592-502b-4d60-adcd-0caa4b5fabdc" containerName="main" containerID="cri-o://54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756" gracePeriod=45
Jul 09 18:22:52.159418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.159393    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:22:52.297254 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297225    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tls-certs\") pod \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") "
Jul 09 18:22:52.297448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnlhs\" (UniqueName: \"kubernetes.io/projected/e13dd592-502b-4d60-adcd-0caa4b5fabdc-kube-api-access-tnlhs\") pod \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") "
Jul 09 18:22:52.297448 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297299    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-model-cache\") pod \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") "
Jul 09 18:22:52.297578 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297468    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tmp-dir\") pod \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") "
Jul 09 18:22:52.297578 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-dshm\") pod \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") "
Jul 09 18:22:52.297578 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297541    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-model-cache" (OuterVolumeSpecName: "model-cache") pod "e13dd592-502b-4d60-adcd-0caa4b5fabdc" (UID: "e13dd592-502b-4d60-adcd-0caa4b5fabdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:52.297737 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-home\") pod \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\" (UID: \"e13dd592-502b-4d60-adcd-0caa4b5fabdc\") "
Jul 09 18:22:52.297801 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297727    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e13dd592-502b-4d60-adcd-0caa4b5fabdc" (UID: "e13dd592-502b-4d60-adcd-0caa4b5fabdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:52.297954 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297928    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-home" (OuterVolumeSpecName: "home") pod "e13dd592-502b-4d60-adcd-0caa4b5fabdc" (UID: "e13dd592-502b-4d60-adcd-0caa4b5fabdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:52.298041 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297959    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:22:52.298041 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.297985    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:22:52.299441 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.299417    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e13dd592-502b-4d60-adcd-0caa4b5fabdc" (UID: "e13dd592-502b-4d60-adcd-0caa4b5fabdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:22:52.299441 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.299426    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13dd592-502b-4d60-adcd-0caa4b5fabdc-kube-api-access-tnlhs" (OuterVolumeSpecName: "kube-api-access-tnlhs") pod "e13dd592-502b-4d60-adcd-0caa4b5fabdc" (UID: "e13dd592-502b-4d60-adcd-0caa4b5fabdc"). InnerVolumeSpecName "kube-api-access-tnlhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:22:52.299441 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.299436    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-dshm" (OuterVolumeSpecName: "dshm") pod "e13dd592-502b-4d60-adcd-0caa4b5fabdc" (UID: "e13dd592-502b-4d60-adcd-0caa4b5fabdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:52.398830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.398795    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13dd592-502b-4d60-adcd-0caa4b5fabdc-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:22:52.398830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.398822    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnlhs\" (UniqueName: \"kubernetes.io/projected/e13dd592-502b-4d60-adcd-0caa4b5fabdc-kube-api-access-tnlhs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:22:52.398830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.398831    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:22:52.398830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.398840    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13dd592-502b-4d60-adcd-0caa4b5fabdc-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:22:52.519712 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.519681    2565 generic.go:356] "Generic (PLEG): container finished" podID="e13dd592-502b-4d60-adcd-0caa4b5fabdc" containerID="54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756" exitCode=0
Jul 09 18:22:52.519881 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.519725    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"
Jul 09 18:22:52.519881 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.519742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk" event={"ID":"e13dd592-502b-4d60-adcd-0caa4b5fabdc","Type":"ContainerDied","Data":"54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756"}
Jul 09 18:22:52.519881 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.519784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk" event={"ID":"e13dd592-502b-4d60-adcd-0caa4b5fabdc","Type":"ContainerDied","Data":"2457afab5f78695308e2e4105cb759399d8d74e360117718558f77beb66308e6"}
Jul 09 18:22:52.519881 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.519807    2565 scope.go:117] "RemoveContainer" containerID="54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756"
Jul 09 18:22:52.528697 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.528678    2565 scope.go:117] "RemoveContainer" containerID="54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756"
Jul 09 18:22:52.528943 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:22:52.528924    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756\": container with ID starting with 54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756 not found: ID does not exist" containerID="54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756"
Jul 09 18:22:52.529009 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.528954    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756"} err="failed to get container status \"54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756\": rpc error: code = NotFound desc = could not find container \"54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756\": container with ID starting with 54262ce7c5a2e622e2e0f3ebd2ba0a8bbf891f937f7d98bd0e7c8ea6df355756 not found: ID does not exist"
Jul 09 18:22:52.541745 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.541723    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"]
Jul 09 18:22:52.545043 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.545023    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk"]
Jul 09 18:22:52.868381 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:22:52.868348    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e13dd592-502b-4d60-adcd-0caa4b5fabdc" path="/var/lib/kubelet/pods/e13dd592-502b-4d60-adcd-0caa4b5fabdc/volumes"
Jul 09 18:23:07.381573 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.381537    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"]
Jul 09 18:23:07.382105 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.382084    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13dd592-502b-4d60-adcd-0caa4b5fabdc" containerName="main"
Jul 09 18:23:07.382205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.382108    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13dd592-502b-4d60-adcd-0caa4b5fabdc" containerName="main"
Jul 09 18:23:07.382205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.382128    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b167043-900e-43a5-ab25-f6fdc9b1e683" containerName="main"
Jul 09 18:23:07.382205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.382136    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b167043-900e-43a5-ab25-f6fdc9b1e683" containerName="main"
Jul 09 18:23:07.382358 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.382234    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b167043-900e-43a5-ab25-f6fdc9b1e683" containerName="main"
Jul 09 18:23:07.382358 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.382245    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13dd592-502b-4d60-adcd-0caa4b5fabdc" containerName="main"
Jul 09 18:23:07.386507 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.386487    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.389588 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.389569    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 09 18:23:07.397436 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.397414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"]
Jul 09 18:23:07.422061 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.422025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.422237 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.422067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.422237 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.422099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.422237 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.422124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88g5\" (UniqueName: \"kubernetes.io/projected/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-kube-api-access-h88g5\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.422237 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.422159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.422237 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.422180    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.523259 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.523221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.523259 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.523267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.523527 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.523288    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.523527 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.523417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88g5\" (UniqueName: \"kubernetes.io/projected/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-kube-api-access-h88g5\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.523527 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.523453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.523527 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.523482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.523754 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.523708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.523818 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.523753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.523858 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.523837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.525594 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.525564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.525854 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.525835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.531890 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.531870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88g5\" (UniqueName: \"kubernetes.io/projected/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-kube-api-access-h88g5\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.696711 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.696686    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:07.826960 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:07.826932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"]
Jul 09 18:23:07.829062 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:23:07.829038    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bde999_9f4f_43d5_abc2_32cfa012dbb9.slice/crio-6462b71b4378051fe6935a7c37e592aac49da1130a451fb8c5f765b0302c84ae WatchSource:0}: Error finding container 6462b71b4378051fe6935a7c37e592aac49da1130a451fb8c5f765b0302c84ae: Status 404 returned error can't find the container with id 6462b71b4378051fe6935a7c37e592aac49da1130a451fb8c5f765b0302c84ae
Jul 09 18:23:08.589007 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:08.588972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk" event={"ID":"a5bde999-9f4f-43d5-abc2-32cfa012dbb9","Type":"ContainerStarted","Data":"ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d"}
Jul 09 18:23:08.589395 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:08.589025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk" event={"ID":"a5bde999-9f4f-43d5-abc2-32cfa012dbb9","Type":"ContainerStarted","Data":"6462b71b4378051fe6935a7c37e592aac49da1130a451fb8c5f765b0302c84ae"}
Jul 09 18:23:08.608102 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:08.608056    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk" podStartSLOduration=1.60804027 podStartE2EDuration="1.60804027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:23:08.605884478 +0000 UTC m=+1108.324147244" watchObservedRunningTime="2026-07-09 18:23:08.60804027 +0000 UTC m=+1108.326303045"
Jul 09 18:23:08.697636 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:08.697601    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:17.697371 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:17.697335    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:17.709618 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:17.709589    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:18.645597 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:18.645567    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:23:46.312228 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:46.312197    2565 scope.go:117] "RemoveContainer" containerID="819a0f82350e3a55c095f3884abd36be109bd95554498caeaa28ffd9592f10d2"
Jul 09 18:23:59.721663 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:59.721626    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"]
Jul 09 18:23:59.731949 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:23:59.731896    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:23:59.731949 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:23:59.731949 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:23:59.731949 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744" cmd=["/bin/sleep","15"]
Jul 09 18:23:59.732290 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:59.731969    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" message="" err=<
Jul 09 18:23:59.732290 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:23:59.732290 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:23:59.732290 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:23:59.732290 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:23:59.731985    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:23:59.732290 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:23:59.732290 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:23:59.732290 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" podUID="6166416a-f536-4de9-bf76-82f8648d9bf0" containerName="main" containerID="cri-o://8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744"
Jul 09 18:23:59.732290 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:59.732022    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" podUID="6166416a-f536-4de9-bf76-82f8648d9bf0" containerName="main" containerID="cri-o://8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744" gracePeriod=60
Jul 09 18:23:59.982104 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:23:59.982035    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:24:00.059769 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.059730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s9zn\" (UniqueName: \"kubernetes.io/projected/6166416a-f536-4de9-bf76-82f8648d9bf0-kube-api-access-2s9zn\") pod \"6166416a-f536-4de9-bf76-82f8648d9bf0\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") "
Jul 09 18:24:00.059967 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.059790    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-tokenizer-uds\") pod \"6166416a-f536-4de9-bf76-82f8648d9bf0\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") "
Jul 09 18:24:00.059967 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.059823    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-kserve-provision-location\") pod \"6166416a-f536-4de9-bf76-82f8648d9bf0\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") "
Jul 09 18:24:00.059967 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.059854    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6166416a-f536-4de9-bf76-82f8648d9bf0-tls-certs\") pod \"6166416a-f536-4de9-bf76-82f8648d9bf0\" (UID: \"6166416a-f536-4de9-bf76-82f8648d9bf0\") "
Jul 09 18:24:00.060098 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.060083    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6166416a-f536-4de9-bf76-82f8648d9bf0" (UID: "6166416a-f536-4de9-bf76-82f8648d9bf0"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:00.060490 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.060462    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6166416a-f536-4de9-bf76-82f8648d9bf0" (UID: "6166416a-f536-4de9-bf76-82f8648d9bf0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:00.061981 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.061951    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6166416a-f536-4de9-bf76-82f8648d9bf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6166416a-f536-4de9-bf76-82f8648d9bf0" (UID: "6166416a-f536-4de9-bf76-82f8648d9bf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:00.062070 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.062034    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6166416a-f536-4de9-bf76-82f8648d9bf0-kube-api-access-2s9zn" (OuterVolumeSpecName: "kube-api-access-2s9zn") pod "6166416a-f536-4de9-bf76-82f8648d9bf0" (UID: "6166416a-f536-4de9-bf76-82f8648d9bf0"). InnerVolumeSpecName "kube-api-access-2s9zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:00.160811 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.160767    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s9zn\" (UniqueName: \"kubernetes.io/projected/6166416a-f536-4de9-bf76-82f8648d9bf0-kube-api-access-2s9zn\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:00.160811 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.160802    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:00.160811 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.160815    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6166416a-f536-4de9-bf76-82f8648d9bf0-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:00.161077 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.160826    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6166416a-f536-4de9-bf76-82f8648d9bf0-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:24:00.844398 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.844364    2565 generic.go:356] "Generic (PLEG): container finished" podID="6166416a-f536-4de9-bf76-82f8648d9bf0" containerID="8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744" exitCode=0
Jul 09 18:24:00.844830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.844418    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"
Jul 09 18:24:00.844830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.844433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" event={"ID":"6166416a-f536-4de9-bf76-82f8648d9bf0","Type":"ContainerDied","Data":"8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744"}
Jul 09 18:24:00.844830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.844497    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz" event={"ID":"6166416a-f536-4de9-bf76-82f8648d9bf0","Type":"ContainerDied","Data":"f95f0690a07f5670dea06c856075f153401558b77f0cdad1e5d0063e39149e51"}
Jul 09 18:24:00.844830 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.844526    2565 scope.go:117] "RemoveContainer" containerID="8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744"
Jul 09 18:24:00.853306 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.853284    2565 scope.go:117] "RemoveContainer" containerID="65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959"
Jul 09 18:24:00.861119 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.861095    2565 scope.go:117] "RemoveContainer" containerID="8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744"
Jul 09 18:24:00.861386 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:24:00.861366    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744\": container with ID starting with 8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744 not found: ID does not exist" containerID="8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744"
Jul 09 18:24:00.861445 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.861396    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744"} err="failed to get container status \"8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744\": rpc error: code = NotFound desc = could not find container \"8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744\": container with ID starting with 8cd6a3ff0df68b832e34c022c9a95240cdbb34a22511a2e9241844d456afc744 not found: ID does not exist"
Jul 09 18:24:00.861445 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.861414    2565 scope.go:117] "RemoveContainer" containerID="65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959"
Jul 09 18:24:00.861634 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:24:00.861617    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959\": container with ID starting with 65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959 not found: ID does not exist" containerID="65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959"
Jul 09 18:24:00.861684 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.861639    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959"} err="failed to get container status \"65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959\": rpc error: code = NotFound desc = could not find container \"65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959\": container with ID starting with 65ff6d7bea40c85baaae8b0d238a603e39875802082188acce0eedef8e158959 not found: ID does not exist"
Jul 09 18:24:00.871068 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.871039    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"]
Jul 09 18:24:00.871068 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:00.871068    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz"]
Jul 09 18:24:02.868081 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:02.868006    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6166416a-f536-4de9-bf76-82f8648d9bf0" path="/var/lib/kubelet/pods/6166416a-f536-4de9-bf76-82f8648d9bf0/volumes"
Jul 09 18:24:23.001659 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.001618    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"]
Jul 09 18:24:23.002134 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.002103    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6166416a-f536-4de9-bf76-82f8648d9bf0" containerName="main"
Jul 09 18:24:23.002134 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.002122    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6166416a-f536-4de9-bf76-82f8648d9bf0" containerName="main"
Jul 09 18:24:23.002243 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.002154    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6166416a-f536-4de9-bf76-82f8648d9bf0" containerName="storage-initializer"
Jul 09 18:24:23.002243 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.002164    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6166416a-f536-4de9-bf76-82f8648d9bf0" containerName="storage-initializer"
Jul 09 18:24:23.002337 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.002280    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6166416a-f536-4de9-bf76-82f8648d9bf0" containerName="main"
Jul 09 18:24:23.005373 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.005350    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.007998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.007976    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:24:23.009182 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.009166    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 09 18:24:23.009233 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.009204    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-twkpj\""
Jul 09 18:24:23.016267 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.016241    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"]
Jul 09 18:24:23.157136 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.157104    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3435f361-8860-4c67-904a-138819a5ac32-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.157136 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.157137    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhlv\" (UniqueName: \"kubernetes.io/projected/3435f361-8860-4c67-904a-138819a5ac32-kube-api-access-5vhlv\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.157339 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.157158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.157339 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.157254    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.257751 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.257665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.257751 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.257738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3435f361-8860-4c67-904a-138819a5ac32-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.257965 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.257773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhlv\" (UniqueName: \"kubernetes.io/projected/3435f361-8860-4c67-904a-138819a5ac32-kube-api-access-5vhlv\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.257965 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.257798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.258180 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.258158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.258216 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.258169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.260292 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.260267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3435f361-8860-4c67-904a-138819a5ac32-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.265740 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.265713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhlv\" (UniqueName: \"kubernetes.io/projected/3435f361-8860-4c67-904a-138819a5ac32-kube-api-access-5vhlv\") pod \"auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.315323 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.315299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:23.453828 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.453788    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"]
Jul 09 18:24:23.456708 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:24:23.456675    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3435f361_8860_4c67_904a_138819a5ac32.slice/crio-d1174f06883cedd750942faaa9327195087db9588b654af0d7f3ea779b7da0c0 WatchSource:0}: Error finding container d1174f06883cedd750942faaa9327195087db9588b654af0d7f3ea779b7da0c0: Status 404 returned error can't find the container with id d1174f06883cedd750942faaa9327195087db9588b654af0d7f3ea779b7da0c0
Jul 09 18:24:23.940525 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.940489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" event={"ID":"3435f361-8860-4c67-904a-138819a5ac32","Type":"ContainerStarted","Data":"6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c"}
Jul 09 18:24:23.940691 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:23.940535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" event={"ID":"3435f361-8860-4c67-904a-138819a5ac32","Type":"ContainerStarted","Data":"d1174f06883cedd750942faaa9327195087db9588b654af0d7f3ea779b7da0c0"}
Jul 09 18:24:24.946554 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:24.946518    2565 generic.go:356] "Generic (PLEG): container finished" podID="3435f361-8860-4c67-904a-138819a5ac32" containerID="6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c" exitCode=0
Jul 09 18:24:24.946977 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:24.946579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" event={"ID":"3435f361-8860-4c67-904a-138819a5ac32","Type":"ContainerDied","Data":"6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c"}
Jul 09 18:24:25.952885 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:25.952851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" event={"ID":"3435f361-8860-4c67-904a-138819a5ac32","Type":"ContainerStarted","Data":"962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141"}
Jul 09 18:24:25.953368 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:25.952959    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:24:25.980736 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:25.980683    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" podStartSLOduration=3.300012626 podStartE2EDuration="3.980663343s" totalImagesPullingTime="0s" totalInitContainerRuntime="680.650717ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:25.978673009 +0000 UTC m=+1185.696935800" watchObservedRunningTime="2026-07-09 18:24:25.980663343 +0000 UTC m=+1185.698926154"
Jul 09 18:24:40.754862 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:40.754835    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:24:40.756017 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:40.755993    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:24:46.331523 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:46.331489    2565 scope.go:117] "RemoveContainer" containerID="08ed4f46727c75d0e8f4ae926844042066670bed290c6b91375f6d471def3215"
Jul 09 18:24:46.339881 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:46.339859    2565 scope.go:117] "RemoveContainer" containerID="ae67cd500758082782dff421264bb013b44518779b572abeb1469ba15b67abc3"
Jul 09 18:24:52.577169 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.577134    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"]
Jul 09 18:24:52.582068 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.582046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.584803 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.584779    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 09 18:24:52.591153 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.591126    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"]
Jul 09 18:24:52.610805 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.610780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.610969 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.610813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5ww\" (UniqueName: \"kubernetes.io/projected/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-kube-api-access-8k5ww\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.610969 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.610837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.610969 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.610945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.611088 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.610995    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.611088 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.611065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.712557 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.712519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.712796 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.712588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.712796 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.712641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.712796 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.712693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.712796 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.712757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.712796 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.712787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5ww\" (UniqueName: \"kubernetes.io/projected/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-kube-api-access-8k5ww\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.713438 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.713406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.713544 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.713430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.713735 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.713715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.715808 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.715784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.715952 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.715869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.722168 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.722144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5ww\" (UniqueName: \"kubernetes.io/projected/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-kube-api-access-8k5ww\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:52.892414 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:52.892328    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:53.028326 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:53.028302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"]
Jul 09 18:24:53.030738 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:24:53.030711    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4da5a44_c293_4c2e_aac4_1c4e02b907ff.slice/crio-de4b2f77a32875915b7aae89abfd7c5a65876dcf145f48710f711c7448ce7857 WatchSource:0}: Error finding container de4b2f77a32875915b7aae89abfd7c5a65876dcf145f48710f711c7448ce7857: Status 404 returned error can't find the container with id de4b2f77a32875915b7aae89abfd7c5a65876dcf145f48710f711c7448ce7857
Jul 09 18:24:53.071061 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:53.071029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4" event={"ID":"a4da5a44-c293-4c2e-aac4-1c4e02b907ff","Type":"ContainerStarted","Data":"de4b2f77a32875915b7aae89abfd7c5a65876dcf145f48710f711c7448ce7857"}
Jul 09 18:24:54.077420 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:54.077383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4" event={"ID":"a4da5a44-c293-4c2e-aac4-1c4e02b907ff","Type":"ContainerStarted","Data":"ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996"}
Jul 09 18:24:54.108986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:54.108938    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4" podStartSLOduration=2.10889806 podStartE2EDuration="2.10889806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:54.108714564 +0000 UTC m=+1213.826977328" watchObservedRunningTime="2026-07-09 18:24:54.10889806 +0000 UTC m=+1213.827160824"
Jul 09 18:24:54.892780 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:54.892741    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:24:56.959534 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:24:56.959506    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:25:02.892992 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:02.892950    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:25:02.905525 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:02.905500    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:25:03.126071 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:03.126041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:25:26.579144 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:26.579105    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"]
Jul 09 18:25:41.599148 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.599020    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4" podUID="a4da5a44-c293-4c2e-aac4-1c4e02b907ff" containerName="main" containerID="cri-o://ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996" gracePeriod=45
Jul 09 18:25:41.851729 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.851669    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:25:41.935948 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.935897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-home\") pod \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") "
Jul 09 18:25:41.935948 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.935948    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tmp-dir\") pod \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") "
Jul 09 18:25:41.936194 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-model-cache\") pod \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") "
Jul 09 18:25:41.936194 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936021    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-dshm\") pod \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") "
Jul 09 18:25:41.936194 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936043    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tls-certs\") pod \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") "
Jul 09 18:25:41.936194 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936082    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k5ww\" (UniqueName: \"kubernetes.io/projected/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-kube-api-access-8k5ww\") pod \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\" (UID: \"a4da5a44-c293-4c2e-aac4-1c4e02b907ff\") "
Jul 09 18:25:41.936397 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936174    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-home" (OuterVolumeSpecName: "home") pod "a4da5a44-c293-4c2e-aac4-1c4e02b907ff" (UID: "a4da5a44-c293-4c2e-aac4-1c4e02b907ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:41.936397 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936267    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4da5a44-c293-4c2e-aac4-1c4e02b907ff" (UID: "a4da5a44-c293-4c2e-aac4-1c4e02b907ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:41.936397 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936278    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4da5a44-c293-4c2e-aac4-1c4e02b907ff" (UID: "a4da5a44-c293-4c2e-aac4-1c4e02b907ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:41.936397 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936345    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:25:41.936397 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936358    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:25:41.936397 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.936367    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:25:41.938168 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.938140    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4da5a44-c293-4c2e-aac4-1c4e02b907ff" (UID: "a4da5a44-c293-4c2e-aac4-1c4e02b907ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:25:41.938168 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.938162    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-dshm" (OuterVolumeSpecName: "dshm") pod "a4da5a44-c293-4c2e-aac4-1c4e02b907ff" (UID: "a4da5a44-c293-4c2e-aac4-1c4e02b907ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:41.938332 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:41.938208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-kube-api-access-8k5ww" (OuterVolumeSpecName: "kube-api-access-8k5ww") pod "a4da5a44-c293-4c2e-aac4-1c4e02b907ff" (UID: "a4da5a44-c293-4c2e-aac4-1c4e02b907ff"). InnerVolumeSpecName "kube-api-access-8k5ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:25:42.037357 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.037316    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:25:42.037357 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.037348    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:25:42.037357 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.037360    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k5ww\" (UniqueName: \"kubernetes.io/projected/a4da5a44-c293-4c2e-aac4-1c4e02b907ff-kube-api-access-8k5ww\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:25:42.289941 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.289894    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4da5a44-c293-4c2e-aac4-1c4e02b907ff" containerID="ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996" exitCode=0
Jul 09 18:25:42.290113 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.289948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4" event={"ID":"a4da5a44-c293-4c2e-aac4-1c4e02b907ff","Type":"ContainerDied","Data":"ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996"}
Jul 09 18:25:42.290113 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.289957    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"
Jul 09 18:25:42.290113 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.290028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4" event={"ID":"a4da5a44-c293-4c2e-aac4-1c4e02b907ff","Type":"ContainerDied","Data":"de4b2f77a32875915b7aae89abfd7c5a65876dcf145f48710f711c7448ce7857"}
Jul 09 18:25:42.290113 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.290052    2565 scope.go:117] "RemoveContainer" containerID="ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996"
Jul 09 18:25:42.298588 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.298570    2565 scope.go:117] "RemoveContainer" containerID="ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996"
Jul 09 18:25:42.298843 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:25:42.298814    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996\": container with ID starting with ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996 not found: ID does not exist" containerID="ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996"
Jul 09 18:25:42.298885 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.298844    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996"} err="failed to get container status \"ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996\": rpc error: code = NotFound desc = could not find container \"ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996\": container with ID starting with ccc8a1f04ed480406efe07ffe71d00310006f4d600a6f4e29142756e718eb996 not found: ID does not exist"
Jul 09 18:25:42.316213 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.316183    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"]
Jul 09 18:25:42.322787 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.322766    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4"]
Jul 09 18:25:42.868277 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:42.868238    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4da5a44-c293-4c2e-aac4-1c4e02b907ff" path="/var/lib/kubelet/pods/a4da5a44-c293-4c2e-aac4-1c4e02b907ff/volumes"
Jul 09 18:25:47.100972 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:25:47.100936    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"]
Jul 09 18:26:02.121809 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.121742    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk" podUID="a5bde999-9f4f-43d5-abc2-32cfa012dbb9" containerName="main" containerID="cri-o://ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d" gracePeriod=45
Jul 09 18:26:02.370958 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.370933    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:26:02.377373 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.377302    2565 generic.go:356] "Generic (PLEG): container finished" podID="a5bde999-9f4f-43d5-abc2-32cfa012dbb9" containerID="ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d" exitCode=0
Jul 09 18:26:02.377517 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.377376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk" event={"ID":"a5bde999-9f4f-43d5-abc2-32cfa012dbb9","Type":"ContainerDied","Data":"ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d"}
Jul 09 18:26:02.377517 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.377387    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"
Jul 09 18:26:02.377517 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.377432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk" event={"ID":"a5bde999-9f4f-43d5-abc2-32cfa012dbb9","Type":"ContainerDied","Data":"6462b71b4378051fe6935a7c37e592aac49da1130a451fb8c5f765b0302c84ae"}
Jul 09 18:26:02.377517 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.377462    2565 scope.go:117] "RemoveContainer" containerID="ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d"
Jul 09 18:26:02.385873 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.385844    2565 scope.go:117] "RemoveContainer" containerID="ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d"
Jul 09 18:26:02.386175 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:26:02.386155    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d\": container with ID starting with ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d not found: ID does not exist" containerID="ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d"
Jul 09 18:26:02.386227 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.386184    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d"} err="failed to get container status \"ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d\": rpc error: code = NotFound desc = could not find container \"ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d\": container with ID starting with ad2e4ceae73c0272a98d38800c2a9aba29afa39357045b8cb9053b547fb68a9d not found: ID does not exist"
Jul 09 18:26:02.518650 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.518615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tmp-dir\") pod \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") "
Jul 09 18:26:02.518650 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.518660    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-home\") pod \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") "
Jul 09 18:26:02.518936 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.518709    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-model-cache\") pod \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") "
Jul 09 18:26:02.518936 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.518749    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tls-certs\") pod \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") "
Jul 09 18:26:02.518936 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.518772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h88g5\" (UniqueName: \"kubernetes.io/projected/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-kube-api-access-h88g5\") pod \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") "
Jul 09 18:26:02.518936 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.518796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-dshm\") pod \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\" (UID: \"a5bde999-9f4f-43d5-abc2-32cfa012dbb9\") "
Jul 09 18:26:02.518936 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.518845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5bde999-9f4f-43d5-abc2-32cfa012dbb9" (UID: "a5bde999-9f4f-43d5-abc2-32cfa012dbb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:02.519452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.519024    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-home" (OuterVolumeSpecName: "home") pod "a5bde999-9f4f-43d5-abc2-32cfa012dbb9" (UID: "a5bde999-9f4f-43d5-abc2-32cfa012dbb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:02.519452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.519034    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5bde999-9f4f-43d5-abc2-32cfa012dbb9" (UID: "a5bde999-9f4f-43d5-abc2-32cfa012dbb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:02.519452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.519126    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:02.519452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.519141    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:02.519452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.519150    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:02.520930 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.520888    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-dshm" (OuterVolumeSpecName: "dshm") pod "a5bde999-9f4f-43d5-abc2-32cfa012dbb9" (UID: "a5bde999-9f4f-43d5-abc2-32cfa012dbb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:02.521026 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.520930    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5bde999-9f4f-43d5-abc2-32cfa012dbb9" (UID: "a5bde999-9f4f-43d5-abc2-32cfa012dbb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:02.521026 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.521008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-kube-api-access-h88g5" (OuterVolumeSpecName: "kube-api-access-h88g5") pod "a5bde999-9f4f-43d5-abc2-32cfa012dbb9" (UID: "a5bde999-9f4f-43d5-abc2-32cfa012dbb9"). InnerVolumeSpecName "kube-api-access-h88g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:02.619565 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.619517    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:02.619565 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.619560    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h88g5\" (UniqueName: \"kubernetes.io/projected/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-kube-api-access-h88g5\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:02.619565 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.619571    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bde999-9f4f-43d5-abc2-32cfa012dbb9-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:02.700493 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.700461    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"]
Jul 09 18:26:02.706069 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.706036    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk"]
Jul 09 18:26:02.868393 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:02.868362    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5bde999-9f4f-43d5-abc2-32cfa012dbb9" path="/var/lib/kubelet/pods/a5bde999-9f4f-43d5-abc2-32cfa012dbb9/volumes"
Jul 09 18:26:19.001031 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.000991    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"]
Jul 09 18:26:19.001492 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.001474    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5bde999-9f4f-43d5-abc2-32cfa012dbb9" containerName="main"
Jul 09 18:26:19.001565 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.001496    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5bde999-9f4f-43d5-abc2-32cfa012dbb9" containerName="main"
Jul 09 18:26:19.001565 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.001513    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4da5a44-c293-4c2e-aac4-1c4e02b907ff" containerName="main"
Jul 09 18:26:19.001565 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.001522    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4da5a44-c293-4c2e-aac4-1c4e02b907ff" containerName="main"
Jul 09 18:26:19.001730 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.001604    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4da5a44-c293-4c2e-aac4-1c4e02b907ff" containerName="main"
Jul 09 18:26:19.001730 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.001619    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5bde999-9f4f-43d5-abc2-32cfa012dbb9" containerName="main"
Jul 09 18:26:19.004704 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.004681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.007413 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.007384    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-pmkxw\""
Jul 09 18:26:19.007989 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.007973    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 09 18:26:19.017007 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.016985    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"]
Jul 09 18:26:19.162398 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.162348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.162398 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.162412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tvd4\" (UniqueName: \"kubernetes.io/projected/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kube-api-access-8tvd4\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.162635 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.162482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.162635 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.162499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.263770 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.263672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.263770 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.263714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.263770 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.263758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.264054 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.263790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvd4\" (UniqueName: \"kubernetes.io/projected/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kube-api-access-8tvd4\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.264162 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.264141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.264232 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.264215    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.266231 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.266212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.273061 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.273040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvd4\" (UniqueName: \"kubernetes.io/projected/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kube-api-access-8tvd4\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.314171 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.314142    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:19.450416 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:19.450382    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"]
Jul 09 18:26:19.453113 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:26:19.453086    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd677fa5_723a_492d_a9f8_1afaaee85ab1.slice/crio-202700e697fa2e92bba757bb71f646de1b2197542c92a3e3aacbb77e57cd9567 WatchSource:0}: Error finding container 202700e697fa2e92bba757bb71f646de1b2197542c92a3e3aacbb77e57cd9567: Status 404 returned error can't find the container with id 202700e697fa2e92bba757bb71f646de1b2197542c92a3e3aacbb77e57cd9567
Jul 09 18:26:20.460599 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:20.460564    2565 generic.go:356] "Generic (PLEG): container finished" podID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" containerID="0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea" exitCode=0
Jul 09 18:26:20.461033 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:20.460626    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd" event={"ID":"bd677fa5-723a-492d-a9f8-1afaaee85ab1","Type":"ContainerDied","Data":"0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea"}
Jul 09 18:26:20.461033 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:20.460665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd" event={"ID":"bd677fa5-723a-492d-a9f8-1afaaee85ab1","Type":"ContainerStarted","Data":"202700e697fa2e92bba757bb71f646de1b2197542c92a3e3aacbb77e57cd9567"}
Jul 09 18:26:21.467217 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:21.467183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd" event={"ID":"bd677fa5-723a-492d-a9f8-1afaaee85ab1","Type":"ContainerStarted","Data":"8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b"}
Jul 09 18:26:21.467604 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:21.467322    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:26:21.492958 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:21.492851    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd" podStartSLOduration=2.854862315 podStartE2EDuration="3.492834303s" totalImagesPullingTime="0s" totalInitContainerRuntime="637.971988ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:21.492325229 +0000 UTC m=+1301.210587994" watchObservedRunningTime="2026-07-09 18:26:21.492834303 +0000 UTC m=+1301.211097096"
Jul 09 18:26:42.908978 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:42.908935    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"]
Jul 09 18:26:42.918421 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:26:42.918383    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:26:42.918421 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:26:42.918421 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:26:42.918421 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141" cmd=["/bin/sleep","15"]
Jul 09 18:26:42.918683 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:42.918436    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" message="" err=<
Jul 09 18:26:42.918683 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:26:42.918683 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:26:42.918683 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:26:42.918683 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:26:42.918448    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:26:42.918683 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:26:42.918683 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:26:42.918683 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" podUID="3435f361-8860-4c67-904a-138819a5ac32" containerName="main" containerID="cri-o://962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141"
Jul 09 18:26:42.918683 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:42.918493    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" podUID="3435f361-8860-4c67-904a-138819a5ac32" containerName="main" containerID="cri-o://962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141" gracePeriod=60
Jul 09 18:26:43.170759 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.170697    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:26:43.271094 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.271063    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3435f361-8860-4c67-904a-138819a5ac32-tls-certs\") pod \"3435f361-8860-4c67-904a-138819a5ac32\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") "
Jul 09 18:26:43.271281 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.271108    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vhlv\" (UniqueName: \"kubernetes.io/projected/3435f361-8860-4c67-904a-138819a5ac32-kube-api-access-5vhlv\") pod \"3435f361-8860-4c67-904a-138819a5ac32\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") "
Jul 09 18:26:43.271281 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.271149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-kserve-provision-location\") pod \"3435f361-8860-4c67-904a-138819a5ac32\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") "
Jul 09 18:26:43.271281 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.271192    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-tokenizer-uds\") pod \"3435f361-8860-4c67-904a-138819a5ac32\" (UID: \"3435f361-8860-4c67-904a-138819a5ac32\") "
Jul 09 18:26:43.271527 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.271505    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3435f361-8860-4c67-904a-138819a5ac32" (UID: "3435f361-8860-4c67-904a-138819a5ac32"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:43.271870 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.271847    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3435f361-8860-4c67-904a-138819a5ac32" (UID: "3435f361-8860-4c67-904a-138819a5ac32"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:43.273265 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.273241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3435f361-8860-4c67-904a-138819a5ac32-kube-api-access-5vhlv" (OuterVolumeSpecName: "kube-api-access-5vhlv") pod "3435f361-8860-4c67-904a-138819a5ac32" (UID: "3435f361-8860-4c67-904a-138819a5ac32"). InnerVolumeSpecName "kube-api-access-5vhlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:43.273374 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.273275    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3435f361-8860-4c67-904a-138819a5ac32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3435f361-8860-4c67-904a-138819a5ac32" (UID: "3435f361-8860-4c67-904a-138819a5ac32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:43.372700 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.372665    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vhlv\" (UniqueName: \"kubernetes.io/projected/3435f361-8860-4c67-904a-138819a5ac32-kube-api-access-5vhlv\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:43.372700 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.372696    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:43.372700 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.372706    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3435f361-8860-4c67-904a-138819a5ac32-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:43.372948 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.372717    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3435f361-8860-4c67-904a-138819a5ac32-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:26:43.565635 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.565601    2565 generic.go:356] "Generic (PLEG): container finished" podID="3435f361-8860-4c67-904a-138819a5ac32" containerID="962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141" exitCode=0
Jul 09 18:26:43.565829 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.565649    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"
Jul 09 18:26:43.565829 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.565680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" event={"ID":"3435f361-8860-4c67-904a-138819a5ac32","Type":"ContainerDied","Data":"962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141"}
Jul 09 18:26:43.565829 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.565728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7" event={"ID":"3435f361-8860-4c67-904a-138819a5ac32","Type":"ContainerDied","Data":"d1174f06883cedd750942faaa9327195087db9588b654af0d7f3ea779b7da0c0"}
Jul 09 18:26:43.565829 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.565748    2565 scope.go:117] "RemoveContainer" containerID="962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141"
Jul 09 18:26:43.575475 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.575459    2565 scope.go:117] "RemoveContainer" containerID="6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c"
Jul 09 18:26:43.582845 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.582828    2565 scope.go:117] "RemoveContainer" containerID="962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141"
Jul 09 18:26:43.583143 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:26:43.583121    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141\": container with ID starting with 962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141 not found: ID does not exist" containerID="962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141"
Jul 09 18:26:43.583254 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.583148    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141"} err="failed to get container status \"962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141\": rpc error: code = NotFound desc = could not find container \"962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141\": container with ID starting with 962b57503caaee8ee2364b3f1d713478efff630aae162ddb3ab33406045c5141 not found: ID does not exist"
Jul 09 18:26:43.583254 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.583165    2565 scope.go:117] "RemoveContainer" containerID="6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c"
Jul 09 18:26:43.583408 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:26:43.583392    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c\": container with ID starting with 6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c not found: ID does not exist" containerID="6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c"
Jul 09 18:26:43.583452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.583415    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c"} err="failed to get container status \"6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c\": rpc error: code = NotFound desc = could not find container \"6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c\": container with ID starting with 6a1a848424a42a9b3d56b66774f8c1e1e1a082a05f55f78e099eb5cc41ee9a4c not found: ID does not exist"
Jul 09 18:26:43.588413 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.588391    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"]
Jul 09 18:26:43.592366 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:43.592345    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7"]
Jul 09 18:26:44.867986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:44.867953    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3435f361-8860-4c67-904a-138819a5ac32" path="/var/lib/kubelet/pods/3435f361-8860-4c67-904a-138819a5ac32/volumes"
Jul 09 18:26:52.473093 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:26:52.473063    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:27:21.178573 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.178540    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"]
Jul 09 18:27:21.179056 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.178852    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3435f361-8860-4c67-904a-138819a5ac32" containerName="main"
Jul 09 18:27:21.179056 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.178863    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3435f361-8860-4c67-904a-138819a5ac32" containerName="main"
Jul 09 18:27:21.179056 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.178875    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3435f361-8860-4c67-904a-138819a5ac32" containerName="storage-initializer"
Jul 09 18:27:21.179056 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.178883    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3435f361-8860-4c67-904a-138819a5ac32" containerName="storage-initializer"
Jul 09 18:27:21.179056 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.178957    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3435f361-8860-4c67-904a-138819a5ac32" containerName="main"
Jul 09 18:27:21.181965 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.181941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.184780 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.184755    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Jul 09 18:27:21.192275 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.192254    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"]
Jul 09 18:27:21.294874 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.294840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.295062 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.294959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.295062 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.294980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td8qs\" (UniqueName: \"kubernetes.io/projected/563043de-1706-4b52-923f-14261beacf03-kube-api-access-td8qs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.295062 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.294998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.295062 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.295019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.295062 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.295036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563043de-1706-4b52-923f-14261beacf03-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.395606 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.395569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.395606 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.395606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8qs\" (UniqueName: \"kubernetes.io/projected/563043de-1706-4b52-923f-14261beacf03-kube-api-access-td8qs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.395828 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.395626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.395828 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.395756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.395828 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.395795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563043de-1706-4b52-923f-14261beacf03-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.396014 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.395841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.396092 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.396069    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.396140 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.396081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.396272 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.396245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.398103 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.398082    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.398209 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.398192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563043de-1706-4b52-923f-14261beacf03-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.404538 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.404514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8qs\" (UniqueName: \"kubernetes.io/projected/563043de-1706-4b52-923f-14261beacf03-kube-api-access-td8qs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.492503 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.492469    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:21.629929 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.629783    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"]
Jul 09 18:27:21.632545 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:27:21.632514    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563043de_1706_4b52_923f_14261beacf03.slice/crio-a8dc883ad3a5b72af8e3cf440e417b18a96ef3efa7da0d52be5dc250bb7b7919 WatchSource:0}: Error finding container a8dc883ad3a5b72af8e3cf440e417b18a96ef3efa7da0d52be5dc250bb7b7919: Status 404 returned error can't find the container with id a8dc883ad3a5b72af8e3cf440e417b18a96ef3efa7da0d52be5dc250bb7b7919
Jul 09 18:27:21.735443 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.735406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9" event={"ID":"563043de-1706-4b52-923f-14261beacf03","Type":"ContainerStarted","Data":"03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440"}
Jul 09 18:27:21.735638 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.735470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9" event={"ID":"563043de-1706-4b52-923f-14261beacf03","Type":"ContainerStarted","Data":"a8dc883ad3a5b72af8e3cf440e417b18a96ef3efa7da0d52be5dc250bb7b7919"}
Jul 09 18:27:21.761942 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:21.761826    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9" podStartSLOduration=0.761807511 podStartE2EDuration="761.807511ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:27:21.760499181 +0000 UTC m=+1361.478761958" watchObservedRunningTime="2026-07-09 18:27:21.761807511 +0000 UTC m=+1361.480070312"
Jul 09 18:27:22.493194 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:22.493161    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:31.492773 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:31.492734    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:31.505779 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:31.505749    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:31.792544 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:31.792455    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:41.833206 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:41.833165    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"]
Jul 09 18:27:56.853116 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:56.853068    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9" podUID="563043de-1706-4b52-923f-14261beacf03" containerName="main" containerID="cri-o://03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440" gracePeriod=45
Jul 09 18:27:57.102192 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.102164    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:57.199810 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.199781    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563043de-1706-4b52-923f-14261beacf03-tls-certs\") pod \"563043de-1706-4b52-923f-14261beacf03\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") "
Jul 09 18:27:57.200025 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.199831    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-home\") pod \"563043de-1706-4b52-923f-14261beacf03\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") "
Jul 09 18:27:57.200025 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.199869    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-model-cache\") pod \"563043de-1706-4b52-923f-14261beacf03\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") "
Jul 09 18:27:57.200025 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.199896    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-tmp-dir\") pod \"563043de-1706-4b52-923f-14261beacf03\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") "
Jul 09 18:27:57.200025 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.199945    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-dshm\") pod \"563043de-1706-4b52-923f-14261beacf03\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") "
Jul 09 18:27:57.200025 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.199970    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td8qs\" (UniqueName: \"kubernetes.io/projected/563043de-1706-4b52-923f-14261beacf03-kube-api-access-td8qs\") pod \"563043de-1706-4b52-923f-14261beacf03\" (UID: \"563043de-1706-4b52-923f-14261beacf03\") "
Jul 09 18:27:57.200309 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.200177    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-home" (OuterVolumeSpecName: "home") pod "563043de-1706-4b52-923f-14261beacf03" (UID: "563043de-1706-4b52-923f-14261beacf03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:57.200309 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.200184    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "563043de-1706-4b52-923f-14261beacf03" (UID: "563043de-1706-4b52-923f-14261beacf03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:57.200309 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.200198    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-model-cache" (OuterVolumeSpecName: "model-cache") pod "563043de-1706-4b52-923f-14261beacf03" (UID: "563043de-1706-4b52-923f-14261beacf03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:57.202154 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.202120    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563043de-1706-4b52-923f-14261beacf03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "563043de-1706-4b52-923f-14261beacf03" (UID: "563043de-1706-4b52-923f-14261beacf03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:27:57.202261 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.202182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-dshm" (OuterVolumeSpecName: "dshm") pod "563043de-1706-4b52-923f-14261beacf03" (UID: "563043de-1706-4b52-923f-14261beacf03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:57.202261 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.202192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563043de-1706-4b52-923f-14261beacf03-kube-api-access-td8qs" (OuterVolumeSpecName: "kube-api-access-td8qs") pod "563043de-1706-4b52-923f-14261beacf03" (UID: "563043de-1706-4b52-923f-14261beacf03"). InnerVolumeSpecName "kube-api-access-td8qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:27:57.301564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.301524    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:27:57.301564 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.301565    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:27:57.301770 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.301578    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:27:57.301770 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.301590    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td8qs\" (UniqueName: \"kubernetes.io/projected/563043de-1706-4b52-923f-14261beacf03-kube-api-access-td8qs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:27:57.301770 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.301603    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563043de-1706-4b52-923f-14261beacf03-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:27:57.301770 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.301632    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/563043de-1706-4b52-923f-14261beacf03-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:27:57.900153 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.900111    2565 generic.go:356] "Generic (PLEG): container finished" podID="563043de-1706-4b52-923f-14261beacf03" containerID="03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440" exitCode=0
Jul 09 18:27:57.900153 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.900155    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"
Jul 09 18:27:57.900609 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.900183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9" event={"ID":"563043de-1706-4b52-923f-14261beacf03","Type":"ContainerDied","Data":"03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440"}
Jul 09 18:27:57.900609 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.900234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9" event={"ID":"563043de-1706-4b52-923f-14261beacf03","Type":"ContainerDied","Data":"a8dc883ad3a5b72af8e3cf440e417b18a96ef3efa7da0d52be5dc250bb7b7919"}
Jul 09 18:27:57.900609 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.900265    2565 scope.go:117] "RemoveContainer" containerID="03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440"
Jul 09 18:27:57.909761 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.909741    2565 scope.go:117] "RemoveContainer" containerID="03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440"
Jul 09 18:27:57.910090 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:27:57.910068    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440\": container with ID starting with 03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440 not found: ID does not exist" containerID="03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440"
Jul 09 18:27:57.910146 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.910101    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440"} err="failed to get container status \"03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440\": rpc error: code = NotFound desc = could not find container \"03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440\": container with ID starting with 03aa415ec797674c2586c15e30d115f1e65a5158eb634afdc6e5668c71d7c440 not found: ID does not exist"
Jul 09 18:27:57.922801 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.922772    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"]
Jul 09 18:27:57.926942 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:57.926917    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9"]
Jul 09 18:27:58.867552 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:27:58.867523    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563043de-1706-4b52-923f-14261beacf03" path="/var/lib/kubelet/pods/563043de-1706-4b52-923f-14261beacf03/volumes"
Jul 09 18:28:22.996048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:22.996008    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"]
Jul 09 18:28:22.996457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:22.996332    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563043de-1706-4b52-923f-14261beacf03" containerName="main"
Jul 09 18:28:22.996457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:22.996343    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="563043de-1706-4b52-923f-14261beacf03" containerName="main"
Jul 09 18:28:22.996457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:22.996406    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="563043de-1706-4b52-923f-14261beacf03" containerName="main"
Jul 09 18:28:22.999452 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:22.999435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.002301 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.002279    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-7f8gw\""
Jul 09 18:28:23.003291 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.003272    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 09 18:28:23.017393 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.017368    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"]
Jul 09 18:28:23.119969 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.119898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.120173 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.119976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfbg\" (UniqueName: \"kubernetes.io/projected/614a8790-4db3-4c6d-99ce-430758523daa-kube-api-access-jkfbg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.120173 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.120000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.120173 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.120132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614a8790-4db3-4c6d-99ce-430758523daa-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.220629 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.220592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614a8790-4db3-4c6d-99ce-430758523daa-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.220831 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.220640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.220831 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.220671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfbg\" (UniqueName: \"kubernetes.io/projected/614a8790-4db3-4c6d-99ce-430758523daa-kube-api-access-jkfbg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.220831 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.220697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.221104 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.221079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.221170 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.221093    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.223161 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.223138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614a8790-4db3-4c6d-99ce-430758523daa-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.232553 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.232529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfbg\" (UniqueName: \"kubernetes.io/projected/614a8790-4db3-4c6d-99ce-430758523daa-kube-api-access-jkfbg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.308408 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.308319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:23.456995 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:23.456969    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"]
Jul 09 18:28:23.459492 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:28:23.459462    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614a8790_4db3_4c6d_99ce_430758523daa.slice/crio-5c2b5f5bd842c831e248e82ed3a4452947f594d9bcb2063dec96a73807c50354 WatchSource:0}: Error finding container 5c2b5f5bd842c831e248e82ed3a4452947f594d9bcb2063dec96a73807c50354: Status 404 returned error can't find the container with id 5c2b5f5bd842c831e248e82ed3a4452947f594d9bcb2063dec96a73807c50354
Jul 09 18:28:24.017419 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:24.017383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" event={"ID":"614a8790-4db3-4c6d-99ce-430758523daa","Type":"ContainerStarted","Data":"3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d"}
Jul 09 18:28:24.017850 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:24.017460    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" event={"ID":"614a8790-4db3-4c6d-99ce-430758523daa","Type":"ContainerStarted","Data":"5c2b5f5bd842c831e248e82ed3a4452947f594d9bcb2063dec96a73807c50354"}
Jul 09 18:28:25.023455 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:25.023423    2565 generic.go:356] "Generic (PLEG): container finished" podID="614a8790-4db3-4c6d-99ce-430758523daa" containerID="3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d" exitCode=0
Jul 09 18:28:25.023851 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:25.023518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" event={"ID":"614a8790-4db3-4c6d-99ce-430758523daa","Type":"ContainerDied","Data":"3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d"}
Jul 09 18:28:26.029958 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:26.029920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" event={"ID":"614a8790-4db3-4c6d-99ce-430758523daa","Type":"ContainerStarted","Data":"5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b"}
Jul 09 18:28:26.030359 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:26.030027    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:28:27.247555 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.247501    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" podStartSLOduration=4.557059676 podStartE2EDuration="5.247480796s" totalImagesPullingTime="0s" totalInitContainerRuntime="690.42112ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:26.068096678 +0000 UTC m=+1425.786359442" watchObservedRunningTime="2026-07-09 18:28:27.247480796 +0000 UTC m=+1426.965743563"
Jul 09 18:28:27.248524 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.248506    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"]
Jul 09 18:28:27.257079 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:28:27.257042    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:28:27.257079 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:28:27.257079 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:28:27.257079 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b" cmd=["/bin/sleep","15"]
Jul 09 18:28:27.257369 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.257102    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd" message="" err=<
Jul 09 18:28:27.257369 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:28:27.257369 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:28:27.257369 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:28:27.257369 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:28:27.257118    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:28:27.257369 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:28:27.257369 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:28:27.257369 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd" podUID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" containerName="main" containerID="cri-o://8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b"
Jul 09 18:28:27.257369 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.257163    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd" podUID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" containerName="main" containerID="cri-o://8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b" gracePeriod=60
Jul 09 18:28:27.505134 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.505071    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:28:27.560205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.560169    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kserve-provision-location\") pod \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") "
Jul 09 18:28:27.560359 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.560224    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tvd4\" (UniqueName: \"kubernetes.io/projected/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kube-api-access-8tvd4\") pod \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") "
Jul 09 18:28:27.560359 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.560284    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tls-certs\") pod \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") "
Jul 09 18:28:27.560359 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.560337    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tokenizer-uds\") pod \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\" (UID: \"bd677fa5-723a-492d-a9f8-1afaaee85ab1\") "
Jul 09 18:28:27.560674 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.560635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "bd677fa5-723a-492d-a9f8-1afaaee85ab1" (UID: "bd677fa5-723a-492d-a9f8-1afaaee85ab1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:27.560985 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.560966    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd677fa5-723a-492d-a9f8-1afaaee85ab1" (UID: "bd677fa5-723a-492d-a9f8-1afaaee85ab1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:27.562309 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.562284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd677fa5-723a-492d-a9f8-1afaaee85ab1" (UID: "bd677fa5-723a-492d-a9f8-1afaaee85ab1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:27.562416 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.562368    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kube-api-access-8tvd4" (OuterVolumeSpecName: "kube-api-access-8tvd4") pod "bd677fa5-723a-492d-a9f8-1afaaee85ab1" (UID: "bd677fa5-723a-492d-a9f8-1afaaee85ab1"). InnerVolumeSpecName "kube-api-access-8tvd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:27.661619 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.661583    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:28:27.661619 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.661613    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tvd4\" (UniqueName: \"kubernetes.io/projected/bd677fa5-723a-492d-a9f8-1afaaee85ab1-kube-api-access-8tvd4\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:28:27.661619 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.661625    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:28:27.661868 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:27.661634    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bd677fa5-723a-492d-a9f8-1afaaee85ab1-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:28:28.040416 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.040389    2565 generic.go:356] "Generic (PLEG): container finished" podID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" containerID="8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b" exitCode=0
Jul 09 18:28:28.040581 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.040440    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"
Jul 09 18:28:28.040581 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.040456    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd" event={"ID":"bd677fa5-723a-492d-a9f8-1afaaee85ab1","Type":"ContainerDied","Data":"8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b"}
Jul 09 18:28:28.040581 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.040496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd" event={"ID":"bd677fa5-723a-492d-a9f8-1afaaee85ab1","Type":"ContainerDied","Data":"202700e697fa2e92bba757bb71f646de1b2197542c92a3e3aacbb77e57cd9567"}
Jul 09 18:28:28.040581 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.040517    2565 scope.go:117] "RemoveContainer" containerID="8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b"
Jul 09 18:28:28.050672 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.050654    2565 scope.go:117] "RemoveContainer" containerID="0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea"
Jul 09 18:28:28.058667 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.058643    2565 scope.go:117] "RemoveContainer" containerID="8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b"
Jul 09 18:28:28.058966 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:28:28.058942    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b\": container with ID starting with 8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b not found: ID does not exist" containerID="8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b"
Jul 09 18:28:28.059058 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.058980    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b"} err="failed to get container status \"8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b\": rpc error: code = NotFound desc = could not find container \"8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b\": container with ID starting with 8401412e78953fc849aea546d8d40ba17d26ddd7d591427b84867aa0952e770b not found: ID does not exist"
Jul 09 18:28:28.059058 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.059006    2565 scope.go:117] "RemoveContainer" containerID="0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea"
Jul 09 18:28:28.059277 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:28:28.059255    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea\": container with ID starting with 0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea not found: ID does not exist" containerID="0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea"
Jul 09 18:28:28.059313 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.059289    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea"} err="failed to get container status \"0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea\": rpc error: code = NotFound desc = could not find container \"0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea\": container with ID starting with 0d44e7fc9334ce0c61630dc2c58a38b8733575e3082c2783a2c31144056e9fea not found: ID does not exist"
Jul 09 18:28:28.069972 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.069947    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"]
Jul 09 18:28:28.076817 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.076793    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd"]
Jul 09 18:28:28.868504 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:28.868470    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" path="/var/lib/kubelet/pods/bd677fa5-723a-492d-a9f8-1afaaee85ab1/volumes"
Jul 09 18:28:57.036374 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:28:57.036343    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:29:07.359887 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.359847    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"]
Jul 09 18:29:07.360275 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.360209    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" containerName="main"
Jul 09 18:29:07.360275 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.360225    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" containerName="main"
Jul 09 18:29:07.360275 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.360238    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" containerName="storage-initializer"
Jul 09 18:29:07.360275 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.360244    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" containerName="storage-initializer"
Jul 09 18:29:07.360403 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.360305    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd677fa5-723a-492d-a9f8-1afaaee85ab1" containerName="main"
Jul 09 18:29:07.365469 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.365449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.368078 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.368058    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 09 18:29:07.368406 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.368389    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-vsbt8\""
Jul 09 18:29:07.379210 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.379182    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"]
Jul 09 18:29:07.392281 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.392254    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmc4\" (UniqueName: \"kubernetes.io/projected/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kube-api-access-sgmc4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.392443 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.392314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.392443 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.392353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.392443 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.392392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.492954 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.492922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.492954 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.492959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.493152 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.493067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.493152 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.493112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmc4\" (UniqueName: \"kubernetes.io/projected/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kube-api-access-sgmc4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.493380 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.493358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.493421 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.493400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.495549 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.495533    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.502324 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.502293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmc4\" (UniqueName: \"kubernetes.io/projected/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kube-api-access-sgmc4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.675165 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.675071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:07.807135 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:07.807086    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"]
Jul 09 18:29:07.809486 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:29:07.809463    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829d3bf0_0cd8_4b3b_a52e_5f1d3e6966bc.slice/crio-1c8bb5b6eea70243be11cee492b4b6693636b6eac211ee4e467de65745559201 WatchSource:0}: Error finding container 1c8bb5b6eea70243be11cee492b4b6693636b6eac211ee4e467de65745559201: Status 404 returned error can't find the container with id 1c8bb5b6eea70243be11cee492b4b6693636b6eac211ee4e467de65745559201
Jul 09 18:29:08.219675 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:08.219641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" event={"ID":"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc","Type":"ContainerStarted","Data":"3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682"}
Jul 09 18:29:08.219871 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:08.219691    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" event={"ID":"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc","Type":"ContainerStarted","Data":"1c8bb5b6eea70243be11cee492b4b6693636b6eac211ee4e467de65745559201"}
Jul 09 18:29:09.225280 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:09.225242    2565 generic.go:356] "Generic (PLEG): container finished" podID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" containerID="3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682" exitCode=0
Jul 09 18:29:09.225280 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:09.225281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" event={"ID":"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc","Type":"ContainerDied","Data":"3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682"}
Jul 09 18:29:10.232005 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:10.231964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" event={"ID":"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc","Type":"ContainerStarted","Data":"3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69"}
Jul 09 18:29:10.232405 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:10.232099    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:29:10.258144 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:10.258098    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" podStartSLOduration=2.611097596 podStartE2EDuration="3.25808299s" totalImagesPullingTime="0s" totalInitContainerRuntime="646.985394ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:10.257110011 +0000 UTC m=+1469.975372800" watchObservedRunningTime="2026-07-09 18:29:10.25808299 +0000 UTC m=+1469.976345753"
Jul 09 18:29:40.780069 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:40.780038    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:29:40.781950 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:40.781926    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:29:41.244075 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:29:41.244050    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:30:38.688731 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:38.688688    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"]
Jul 09 18:30:38.697605 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:30:38.697563    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:30:38.697605 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:30:38.697605 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:30:38.697605 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b" cmd=["/bin/sleep","15"]
Jul 09 18:30:38.697855 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:38.697619    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" message="" err=<
Jul 09 18:30:38.697855 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:30:38.697855 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:30:38.697855 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:30:38.697855 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:30:38.697631    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:30:38.697855 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:30:38.697855 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:30:38.697855 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" podUID="614a8790-4db3-4c6d-99ce-430758523daa" containerName="main" containerID="cri-o://5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b"
Jul 09 18:30:38.697855 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:38.697673    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" podUID="614a8790-4db3-4c6d-99ce-430758523daa" containerName="main" containerID="cri-o://5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b" gracePeriod=60
Jul 09 18:30:38.954639 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:38.954610    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:30:39.069955 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.069883    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkfbg\" (UniqueName: \"kubernetes.io/projected/614a8790-4db3-4c6d-99ce-430758523daa-kube-api-access-jkfbg\") pod \"614a8790-4db3-4c6d-99ce-430758523daa\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") "
Jul 09 18:30:39.070155 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.070058    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-tokenizer-uds\") pod \"614a8790-4db3-4c6d-99ce-430758523daa\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") "
Jul 09 18:30:39.070155 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.070113    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-kserve-provision-location\") pod \"614a8790-4db3-4c6d-99ce-430758523daa\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") "
Jul 09 18:30:39.070155 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.070140    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614a8790-4db3-4c6d-99ce-430758523daa-tls-certs\") pod \"614a8790-4db3-4c6d-99ce-430758523daa\" (UID: \"614a8790-4db3-4c6d-99ce-430758523daa\") "
Jul 09 18:30:39.070380 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.070349    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "614a8790-4db3-4c6d-99ce-430758523daa" (UID: "614a8790-4db3-4c6d-99ce-430758523daa"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:39.070850 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.070822    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "614a8790-4db3-4c6d-99ce-430758523daa" (UID: "614a8790-4db3-4c6d-99ce-430758523daa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:39.071944 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.071921    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614a8790-4db3-4c6d-99ce-430758523daa-kube-api-access-jkfbg" (OuterVolumeSpecName: "kube-api-access-jkfbg") pod "614a8790-4db3-4c6d-99ce-430758523daa" (UID: "614a8790-4db3-4c6d-99ce-430758523daa"). InnerVolumeSpecName "kube-api-access-jkfbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:30:39.072400 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.072381    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614a8790-4db3-4c6d-99ce-430758523daa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "614a8790-4db3-4c6d-99ce-430758523daa" (UID: "614a8790-4db3-4c6d-99ce-430758523daa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:30:39.171715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.171682    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:30:39.171715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.171711    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/614a8790-4db3-4c6d-99ce-430758523daa-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:30:39.171715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.171721    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614a8790-4db3-4c6d-99ce-430758523daa-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:30:39.171993 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.171732    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkfbg\" (UniqueName: \"kubernetes.io/projected/614a8790-4db3-4c6d-99ce-430758523daa-kube-api-access-jkfbg\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:30:39.631943 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.631881    2565 generic.go:356] "Generic (PLEG): container finished" podID="614a8790-4db3-4c6d-99ce-430758523daa" containerID="5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b" exitCode=0
Jul 09 18:30:39.632117 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.631965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" event={"ID":"614a8790-4db3-4c6d-99ce-430758523daa","Type":"ContainerDied","Data":"5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b"}
Jul 09 18:30:39.632117 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.631982    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"
Jul 09 18:30:39.632117 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.632047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj" event={"ID":"614a8790-4db3-4c6d-99ce-430758523daa","Type":"ContainerDied","Data":"5c2b5f5bd842c831e248e82ed3a4452947f594d9bcb2063dec96a73807c50354"}
Jul 09 18:30:39.632117 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.632076    2565 scope.go:117] "RemoveContainer" containerID="5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b"
Jul 09 18:30:39.641581 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.641561    2565 scope.go:117] "RemoveContainer" containerID="3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d"
Jul 09 18:30:39.649764 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.649743    2565 scope.go:117] "RemoveContainer" containerID="5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b"
Jul 09 18:30:39.650112 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:30:39.650086    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b\": container with ID starting with 5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b not found: ID does not exist" containerID="5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b"
Jul 09 18:30:39.650200 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.650123    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b"} err="failed to get container status \"5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b\": rpc error: code = NotFound desc = could not find container \"5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b\": container with ID starting with 5533d80d705c4d0d8ee3e165c4b7200a4d51301c196410d30949a1a0eb548f9b not found: ID does not exist"
Jul 09 18:30:39.650200 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.650148    2565 scope.go:117] "RemoveContainer" containerID="3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d"
Jul 09 18:30:39.650426 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:30:39.650402    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d\": container with ID starting with 3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d not found: ID does not exist" containerID="3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d"
Jul 09 18:30:39.650493 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.650432    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d"} err="failed to get container status \"3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d\": rpc error: code = NotFound desc = could not find container \"3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d\": container with ID starting with 3de6c6566124e38d1f4a96f529c34927d0769725063fd9c574b742a6cba0501d not found: ID does not exist"
Jul 09 18:30:39.654273 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.654248    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"]
Jul 09 18:30:39.657834 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:39.657801    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj"]
Jul 09 18:30:40.868188 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:40.868158    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="614a8790-4db3-4c6d-99ce-430758523daa" path="/var/lib/kubelet/pods/614a8790-4db3-4c6d-99ce-430758523daa/volumes"
Jul 09 18:30:51.287627 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.287591    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"]
Jul 09 18:30:51.296802 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:30:51.296766    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:30:51.296802 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:30:51.296802 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:30:51.296802 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69" cmd=["/bin/sleep","15"]
Jul 09 18:30:51.297148 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.296819    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" message="" err=<
Jul 09 18:30:51.297148 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:30:51.297148 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:30:51.297148 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:30:51.297148 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:30:51.296831    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:30:51.297148 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:30:51.297148 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:30:51.297148 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" podUID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" containerName="main" containerID="cri-o://3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69"
Jul 09 18:30:51.297148 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.296881    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" podUID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" containerName="main" containerID="cri-o://3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69" gracePeriod=60
Jul 09 18:30:51.551265 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.551206    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:30:51.578897 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.578860    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgmc4\" (UniqueName: \"kubernetes.io/projected/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kube-api-access-sgmc4\") pod \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") "
Jul 09 18:30:51.579139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.578941    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tokenizer-uds\") pod \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") "
Jul 09 18:30:51.579139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.578980    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kserve-provision-location\") pod \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") "
Jul 09 18:30:51.579139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.579015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tls-certs\") pod \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\" (UID: \"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc\") "
Jul 09 18:30:51.579315 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.579254    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" (UID: "829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:51.579715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.579692    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" (UID: "829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:51.581150 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.581119    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" (UID: "829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:30:51.581518 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.581496    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kube-api-access-sgmc4" (OuterVolumeSpecName: "kube-api-access-sgmc4") pod "829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" (UID: "829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc"). InnerVolumeSpecName "kube-api-access-sgmc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:30:51.679589 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.679546    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgmc4\" (UniqueName: \"kubernetes.io/projected/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kube-api-access-sgmc4\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:30:51.679589 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.679583    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:30:51.679589 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.679594    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:30:51.679822 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.679604    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:30:51.686076 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.686046    2565 generic.go:356] "Generic (PLEG): container finished" podID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" containerID="3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69" exitCode=0
Jul 09 18:30:51.686228 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.686088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" event={"ID":"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc","Type":"ContainerDied","Data":"3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69"}
Jul 09 18:30:51.686228 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.686095    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"
Jul 09 18:30:51.686228 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.686170    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm" event={"ID":"829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc","Type":"ContainerDied","Data":"1c8bb5b6eea70243be11cee492b4b6693636b6eac211ee4e467de65745559201"}
Jul 09 18:30:51.686228 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.686198    2565 scope.go:117] "RemoveContainer" containerID="3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69"
Jul 09 18:30:51.695224 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.695202    2565 scope.go:117] "RemoveContainer" containerID="3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682"
Jul 09 18:30:51.702544 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.702526    2565 scope.go:117] "RemoveContainer" containerID="3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69"
Jul 09 18:30:51.702803 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:30:51.702783    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69\": container with ID starting with 3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69 not found: ID does not exist" containerID="3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69"
Jul 09 18:30:51.702854 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.702811    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69"} err="failed to get container status \"3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69\": rpc error: code = NotFound desc = could not find container \"3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69\": container with ID starting with 3feaf9c10f471cf9b0c9377cdd2a6ce39b28c38603048fb3da269fef68fbdc69 not found: ID does not exist"
Jul 09 18:30:51.702854 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.702830    2565 scope.go:117] "RemoveContainer" containerID="3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682"
Jul 09 18:30:51.703105 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:30:51.703069    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682\": container with ID starting with 3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682 not found: ID does not exist" containerID="3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682"
Jul 09 18:30:51.703105 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.703087    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682"} err="failed to get container status \"3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682\": rpc error: code = NotFound desc = could not find container \"3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682\": container with ID starting with 3646ee4b41fdb7163436e06b29cd1667458cef8cba1e72428c3c0df1744f0682 not found: ID does not exist"
Jul 09 18:30:51.708342 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.708320    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"]
Jul 09 18:30:51.712998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:51.712972    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm"]
Jul 09 18:30:52.867420 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:30:52.867384    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" path="/var/lib/kubelet/pods/829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc/volumes"
Jul 09 18:31:04.549092 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549052    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"]
Jul 09 18:31:04.549612 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549593    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" containerName="storage-initializer"
Jul 09 18:31:04.549705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549617    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" containerName="storage-initializer"
Jul 09 18:31:04.549705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549652    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614a8790-4db3-4c6d-99ce-430758523daa" containerName="storage-initializer"
Jul 09 18:31:04.549705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549662    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="614a8790-4db3-4c6d-99ce-430758523daa" containerName="storage-initializer"
Jul 09 18:31:04.549705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549673    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614a8790-4db3-4c6d-99ce-430758523daa" containerName="main"
Jul 09 18:31:04.549705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549683    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="614a8790-4db3-4c6d-99ce-430758523daa" containerName="main"
Jul 09 18:31:04.549705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549692    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" containerName="main"
Jul 09 18:31:04.549705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549700    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" containerName="main"
Jul 09 18:31:04.550079 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549802    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="829d3bf0-0cd8-4b3b-a52e-5f1d3e6966bc" containerName="main"
Jul 09 18:31:04.550079 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.549819    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="614a8790-4db3-4c6d-99ce-430758523daa" containerName="main"
Jul 09 18:31:04.554381 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.554358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.558119 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.558098    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:31:04.560581 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.560557    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 09 18:31:04.560717 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.560611    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-s8z92\""
Jul 09 18:31:04.560717 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.560657    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:31:04.560995 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.560974    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:31:04.564220 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.562858    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-zpp4m\""
Jul 09 18:31:04.567559 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.567536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f07daa-e620-4c89-8072-7c19132e8de3-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.567664 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.567593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.567745 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.567720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzdvx\" (UniqueName: \"kubernetes.io/projected/b1f07daa-e620-4c89-8072-7c19132e8de3-kube-api-access-pzdvx\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.568048 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.568021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.570088 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.570068    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"]
Jul 09 18:31:04.669251 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.669214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzdvx\" (UniqueName: \"kubernetes.io/projected/b1f07daa-e620-4c89-8072-7c19132e8de3-kube-api-access-pzdvx\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.669424 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.669256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.669424 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.669304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f07daa-e620-4c89-8072-7c19132e8de3-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.669424 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.669331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.669767 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.669739    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.669866 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.669750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.671846 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.671822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f07daa-e620-4c89-8072-7c19132e8de3-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.680569 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.680535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzdvx\" (UniqueName: \"kubernetes.io/projected/b1f07daa-e620-4c89-8072-7c19132e8de3-kube-api-access-pzdvx\") pod \"custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:04.864930 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:04.864820    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:05.003470 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:05.003444    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"]
Jul 09 18:31:05.005557 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:31:05.005530    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f07daa_e620_4c89_8072_7c19132e8de3.slice/crio-1b3d00edfcfa38becc914f82928b74e7a0a1a3d46a4ebeb2b55feb526a2ba491 WatchSource:0}: Error finding container 1b3d00edfcfa38becc914f82928b74e7a0a1a3d46a4ebeb2b55feb526a2ba491: Status 404 returned error can't find the container with id 1b3d00edfcfa38becc914f82928b74e7a0a1a3d46a4ebeb2b55feb526a2ba491
Jul 09 18:31:05.747478 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:05.747409    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1f07daa-e620-4c89-8072-7c19132e8de3" containerID="4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3" exitCode=0
Jul 09 18:31:05.747925 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:05.747497    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g" event={"ID":"b1f07daa-e620-4c89-8072-7c19132e8de3","Type":"ContainerDied","Data":"4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3"}
Jul 09 18:31:05.747925 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:05.747553    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g" event={"ID":"b1f07daa-e620-4c89-8072-7c19132e8de3","Type":"ContainerStarted","Data":"1b3d00edfcfa38becc914f82928b74e7a0a1a3d46a4ebeb2b55feb526a2ba491"}
Jul 09 18:31:06.753787 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:06.753746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g" event={"ID":"b1f07daa-e620-4c89-8072-7c19132e8de3","Type":"ContainerStarted","Data":"7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465"}
Jul 09 18:31:06.754220 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:06.753817    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:06.776056 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:06.776000    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g" podStartSLOduration=2.137782932 podStartE2EDuration="2.775983353s" totalImagesPullingTime="0s" totalInitContainerRuntime="638.200421ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:31:06.774980168 +0000 UTC m=+1586.493242945" watchObservedRunningTime="2026-07-09 18:31:06.775983353 +0000 UTC m=+1586.494246180"
Jul 09 18:31:14.738612 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:14.738571    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"]
Jul 09 18:31:14.741861 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:14.741845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:14.771740 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:14.771708    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"]
Jul 09 18:31:14.852541 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:14.852512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5qc\" (UniqueName: \"kubernetes.io/projected/d1a50ed2-da02-4097-8351-d11e25d71cb9-kube-api-access-qk5qc\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:14.952996 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:14.952957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5qc\" (UniqueName: \"kubernetes.io/projected/d1a50ed2-da02-4097-8351-d11e25d71cb9-kube-api-access-qk5qc\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:14.963803 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:14.963781    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5qc\" (UniqueName: \"kubernetes.io/projected/d1a50ed2-da02-4097-8351-d11e25d71cb9-kube-api-access-qk5qc\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:17.170008 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:17.169968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:17.191021 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:17.190990    2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:17.270427 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:17.270389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:17.270601 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:17.270539    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:17.274115 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:17.274092    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") DevicePath \"csi-55bd14af576e77d6436311c1b9c2889f6970de0650f144b747e86982125a0cd9\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:17.514517 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:17.514476    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/5194e7d566e20e82448dd728d8bd83db3a63ffe4ee741a7ad074e0f3f76777f7/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:17.528543 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:17.528514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"e2e-pvc-model-download-hx52p\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:17.750663 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:17.750568    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:17.884238 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:17.884214    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"]
Jul 09 18:31:17.886264 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:31:17.886237    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1a50ed2_da02_4097_8351_d11e25d71cb9.slice/crio-827fbe2e074f1d52c13fae6f8dfc5e90d6c35b9d88f9f36fe82769a5db6e4f63 WatchSource:0}: Error finding container 827fbe2e074f1d52c13fae6f8dfc5e90d6c35b9d88f9f36fe82769a5db6e4f63: Status 404 returned error can't find the container with id 827fbe2e074f1d52c13fae6f8dfc5e90d6c35b9d88f9f36fe82769a5db6e4f63
Jul 09 18:31:18.813024 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:18.812981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p" event={"ID":"d1a50ed2-da02-4097-8351-d11e25d71cb9","Type":"ContainerStarted","Data":"982fd4bb11846f8827eaec352c1ffd71288a6c22b45a90660152218571cd03f6"}
Jul 09 18:31:18.813732 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:18.813704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p" event={"ID":"d1a50ed2-da02-4097-8351-d11e25d71cb9","Type":"ContainerStarted","Data":"827fbe2e074f1d52c13fae6f8dfc5e90d6c35b9d88f9f36fe82769a5db6e4f63"}
Jul 09 18:31:24.839479 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:24.839393    2565 generic.go:356] "Generic (PLEG): container finished" podID="d1a50ed2-da02-4097-8351-d11e25d71cb9" containerID="982fd4bb11846f8827eaec352c1ffd71288a6c22b45a90660152218571cd03f6" exitCode=0
Jul 09 18:31:24.839479 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:24.839451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p" event={"ID":"d1a50ed2-da02-4097-8351-d11e25d71cb9","Type":"ContainerDied","Data":"982fd4bb11846f8827eaec352c1ffd71288a6c22b45a90660152218571cd03f6"}
Jul 09 18:31:25.990559 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:25.990535    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:26.148753 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.148648    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk5qc\" (UniqueName: \"kubernetes.io/projected/d1a50ed2-da02-4097-8351-d11e25d71cb9-kube-api-access-qk5qc\") pod \"d1a50ed2-da02-4097-8351-d11e25d71cb9\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") "
Jul 09 18:31:26.148948 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.148818    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"d1a50ed2-da02-4097-8351-d11e25d71cb9\" (UID: \"d1a50ed2-da02-4097-8351-d11e25d71cb9\") "
Jul 09 18:31:26.150998 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.150965    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a50ed2-da02-4097-8351-d11e25d71cb9-kube-api-access-qk5qc" (OuterVolumeSpecName: "kube-api-access-qk5qc") pod "d1a50ed2-da02-4097-8351-d11e25d71cb9" (UID: "d1a50ed2-da02-4097-8351-d11e25d71cb9"). InnerVolumeSpecName "kube-api-access-qk5qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:31:26.152699 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.152675    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564" (OuterVolumeSpecName: "model-storage") pod "d1a50ed2-da02-4097-8351-d11e25d71cb9" (UID: "d1a50ed2-da02-4097-8351-d11e25d71cb9"). InnerVolumeSpecName "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:31:26.250393 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.250346    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") on node \"ip-10-0-129-87.ec2.internal\" "
Jul 09 18:31:26.250393 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.250391    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk5qc\" (UniqueName: \"kubernetes.io/projected/d1a50ed2-da02-4097-8351-d11e25d71cb9-kube-api-access-qk5qc\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:31:26.366518 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.366496    2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564") on node "ip-10-0-129-87.ec2.internal"
Jul 09 18:31:26.452124 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.452095    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"csi-55bd14af576e77d6436311c1b9c2889f6970de0650f144b747e86982125a0cd9\""
Jul 09 18:31:26.849563 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.849487    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p"
Jul 09 18:31:26.849713 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.849484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-hx52p" event={"ID":"d1a50ed2-da02-4097-8351-d11e25d71cb9","Type":"ContainerDied","Data":"827fbe2e074f1d52c13fae6f8dfc5e90d6c35b9d88f9f36fe82769a5db6e4f63"}
Jul 09 18:31:26.849713 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:26.849611    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="827fbe2e074f1d52c13fae6f8dfc5e90d6c35b9d88f9f36fe82769a5db6e4f63"
Jul 09 18:31:36.915571 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.915475    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"]
Jul 09 18:31:36.916039 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.916019    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a50ed2-da02-4097-8351-d11e25d71cb9" containerName="storage-initializer"
Jul 09 18:31:36.916101 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.916042    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a50ed2-da02-4097-8351-d11e25d71cb9" containerName="storage-initializer"
Jul 09 18:31:36.916137 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.916126    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a50ed2-da02-4097-8351-d11e25d71cb9" containerName="storage-initializer"
Jul 09 18:31:36.919705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.919681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:36.923548 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.923525    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-rrhx2\""
Jul 09 18:31:36.923700 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.923676    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 09 18:31:36.932418 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.932396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"]
Jul 09 18:31:36.933996 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.933972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f64bm\" (UniqueName: \"kubernetes.io/projected/0b12fcf9-7192-4135-a99e-9d03a833fd7b-kube-api-access-f64bm\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5\" (UID: \"0b12fcf9-7192-4135-a99e-9d03a833fd7b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:36.934101 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.934015    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b12fcf9-7192-4135-a99e-9d03a833fd7b-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5\" (UID: \"0b12fcf9-7192-4135-a99e-9d03a833fd7b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:36.934148 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:36.934104    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b12fcf9-7192-4135-a99e-9d03a833fd7b-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5\" (UID: \"0b12fcf9-7192-4135-a99e-9d03a833fd7b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:37.034731 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.034692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b12fcf9-7192-4135-a99e-9d03a833fd7b-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5\" (UID: \"0b12fcf9-7192-4135-a99e-9d03a833fd7b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:37.034931 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.034749    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64bm\" (UniqueName: \"kubernetes.io/projected/0b12fcf9-7192-4135-a99e-9d03a833fd7b-kube-api-access-f64bm\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5\" (UID: \"0b12fcf9-7192-4135-a99e-9d03a833fd7b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:37.034931 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.034776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b12fcf9-7192-4135-a99e-9d03a833fd7b-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5\" (UID: \"0b12fcf9-7192-4135-a99e-9d03a833fd7b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:37.035187 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.035163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b12fcf9-7192-4135-a99e-9d03a833fd7b-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5\" (UID: \"0b12fcf9-7192-4135-a99e-9d03a833fd7b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:37.037263 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.037238    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b12fcf9-7192-4135-a99e-9d03a833fd7b-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5\" (UID: \"0b12fcf9-7192-4135-a99e-9d03a833fd7b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:37.043386 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.043357    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f64bm\" (UniqueName: \"kubernetes.io/projected/0b12fcf9-7192-4135-a99e-9d03a833fd7b-kube-api-access-f64bm\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5\" (UID: \"0b12fcf9-7192-4135-a99e-9d03a833fd7b\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:37.229391 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.229352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:37.360684 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.360657    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"]
Jul 09 18:31:37.362999 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:31:37.362974    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b12fcf9_7192_4135_a99e_9d03a833fd7b.slice/crio-f3d37e5384b75d2cc8815a9982f07e0f36fe2df8b3360b16b8e4bab396c5eef4 WatchSource:0}: Error finding container f3d37e5384b75d2cc8815a9982f07e0f36fe2df8b3360b16b8e4bab396c5eef4: Status 404 returned error can't find the container with id f3d37e5384b75d2cc8815a9982f07e0f36fe2df8b3360b16b8e4bab396c5eef4
Jul 09 18:31:37.759929 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.759875    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:31:37.902718 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.902685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5" event={"ID":"0b12fcf9-7192-4135-a99e-9d03a833fd7b","Type":"ContainerStarted","Data":"150dc5eb845cd9ec2c4a8add38fe6e3850be4c745b09a94bc1be33266e7521bf"}
Jul 09 18:31:37.902885 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.902734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5" event={"ID":"0b12fcf9-7192-4135-a99e-9d03a833fd7b","Type":"ContainerStarted","Data":"f3d37e5384b75d2cc8815a9982f07e0f36fe2df8b3360b16b8e4bab396c5eef4"}
Jul 09 18:31:37.902885 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.902797    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:31:37.922637 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:31:37.922590    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5" podStartSLOduration=1.9225739160000002 podStartE2EDuration="1.922573916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:31:37.920964508 +0000 UTC m=+1617.639227272" watchObservedRunningTime="2026-07-09 18:31:37.922573916 +0000 UTC m=+1617.640836699"
Jul 09 18:32:08.908734 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:32:08.908705    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5"
Jul 09 18:33:49.582735 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.582671    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"]
Jul 09 18:33:49.591620 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:33:49.591584    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:33:49.591620 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:33:49.591620 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:33:49.591620 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465" cmd=["/bin/sleep","15"]
Jul 09 18:33:49.591846 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.591641    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g" message="" err=<
Jul 09 18:33:49.591846 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:33:49.591846 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:33:49.591846 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:33:49.591846 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:33:49.591654    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:33:49.591846 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:33:49.591846 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:33:49.591846 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g" podUID="b1f07daa-e620-4c89-8072-7c19132e8de3" containerName="main" containerID="cri-o://7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465"
Jul 09 18:33:49.591846 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.591697    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g" podUID="b1f07daa-e620-4c89-8072-7c19132e8de3" containerName="main" containerID="cri-o://7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465" gracePeriod=60
Jul 09 18:33:49.840263 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.840201    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:33:49.980126 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.980090    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-tokenizer-uds\") pod \"b1f07daa-e620-4c89-8072-7c19132e8de3\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") "
Jul 09 18:33:49.980126 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.980129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-kserve-provision-location\") pod \"b1f07daa-e620-4c89-8072-7c19132e8de3\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") "
Jul 09 18:33:49.980333 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.980166    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzdvx\" (UniqueName: \"kubernetes.io/projected/b1f07daa-e620-4c89-8072-7c19132e8de3-kube-api-access-pzdvx\") pod \"b1f07daa-e620-4c89-8072-7c19132e8de3\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") "
Jul 09 18:33:49.980333 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.980200    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f07daa-e620-4c89-8072-7c19132e8de3-tls-certs\") pod \"b1f07daa-e620-4c89-8072-7c19132e8de3\" (UID: \"b1f07daa-e620-4c89-8072-7c19132e8de3\") "
Jul 09 18:33:49.980421 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.980382    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b1f07daa-e620-4c89-8072-7c19132e8de3" (UID: "b1f07daa-e620-4c89-8072-7c19132e8de3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:49.980505 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.980487    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:33:49.980814 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.980782    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1f07daa-e620-4c89-8072-7c19132e8de3" (UID: "b1f07daa-e620-4c89-8072-7c19132e8de3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:49.982266 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.982241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f07daa-e620-4c89-8072-7c19132e8de3-kube-api-access-pzdvx" (OuterVolumeSpecName: "kube-api-access-pzdvx") pod "b1f07daa-e620-4c89-8072-7c19132e8de3" (UID: "b1f07daa-e620-4c89-8072-7c19132e8de3"). InnerVolumeSpecName "kube-api-access-pzdvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:33:49.982364 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:49.982285    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f07daa-e620-4c89-8072-7c19132e8de3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1f07daa-e620-4c89-8072-7c19132e8de3" (UID: "b1f07daa-e620-4c89-8072-7c19132e8de3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:33:50.081109 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.081059    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f07daa-e620-4c89-8072-7c19132e8de3-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:33:50.081109 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.081103    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzdvx\" (UniqueName: \"kubernetes.io/projected/b1f07daa-e620-4c89-8072-7c19132e8de3-kube-api-access-pzdvx\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:33:50.081109 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.081113    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f07daa-e620-4c89-8072-7c19132e8de3-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:33:50.492040 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.492002    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1f07daa-e620-4c89-8072-7c19132e8de3" containerID="7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465" exitCode=0
Jul 09 18:33:50.492232 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.492067    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"
Jul 09 18:33:50.492232 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.492091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g" event={"ID":"b1f07daa-e620-4c89-8072-7c19132e8de3","Type":"ContainerDied","Data":"7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465"}
Jul 09 18:33:50.492232 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.492135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g" event={"ID":"b1f07daa-e620-4c89-8072-7c19132e8de3","Type":"ContainerDied","Data":"1b3d00edfcfa38becc914f82928b74e7a0a1a3d46a4ebeb2b55feb526a2ba491"}
Jul 09 18:33:50.492232 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.492158    2565 scope.go:117] "RemoveContainer" containerID="7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465"
Jul 09 18:33:50.502044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.502025    2565 scope.go:117] "RemoveContainer" containerID="4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3"
Jul 09 18:33:50.510224 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.510207    2565 scope.go:117] "RemoveContainer" containerID="7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465"
Jul 09 18:33:50.510502 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:33:50.510479    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465\": container with ID starting with 7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465 not found: ID does not exist" containerID="7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465"
Jul 09 18:33:50.510551 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.510512    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465"} err="failed to get container status \"7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465\": rpc error: code = NotFound desc = could not find container \"7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465\": container with ID starting with 7fad2e1f6e381f45ea36f465bfe0817c5f03be0ab3b1e0bb8cbadc965b3e3465 not found: ID does not exist"
Jul 09 18:33:50.510551 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.510530    2565 scope.go:117] "RemoveContainer" containerID="4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3"
Jul 09 18:33:50.510769 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:33:50.510751    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3\": container with ID starting with 4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3 not found: ID does not exist" containerID="4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3"
Jul 09 18:33:50.510806 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.510776    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3"} err="failed to get container status \"4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3\": rpc error: code = NotFound desc = could not find container \"4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3\": container with ID starting with 4b389e9f86b71ca828d917b17f8dc6d3b94af887b9185db4351f522cd17a41a3 not found: ID does not exist"
Jul 09 18:33:50.519075 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.519047    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"]
Jul 09 18:33:50.523335 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.523310    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g"]
Jul 09 18:33:50.868106 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:33:50.868028    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f07daa-e620-4c89-8072-7c19132e8de3" path="/var/lib/kubelet/pods/b1f07daa-e620-4c89-8072-7c19132e8de3/volumes"
Jul 09 18:34:12.832238 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.832206    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"]
Jul 09 18:34:12.832733 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.832539    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f07daa-e620-4c89-8072-7c19132e8de3" containerName="storage-initializer"
Jul 09 18:34:12.832733 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.832551    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f07daa-e620-4c89-8072-7c19132e8de3" containerName="storage-initializer"
Jul 09 18:34:12.832733 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.832558    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f07daa-e620-4c89-8072-7c19132e8de3" containerName="main"
Jul 09 18:34:12.832733 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.832564    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f07daa-e620-4c89-8072-7c19132e8de3" containerName="main"
Jul 09 18:34:12.832733 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.832621    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f07daa-e620-4c89-8072-7c19132e8de3" containerName="main"
Jul 09 18:34:12.835853 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.835833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.839050 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.839022    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:34:12.839158 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.839106    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-7sv49\""
Jul 09 18:34:12.851054 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.851030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"]
Jul 09 18:34:12.877303 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.877269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.877470 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.877315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.877470 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.877341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.877470 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.877389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.877470 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.877420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.877614 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.877508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrh7h\" (UniqueName: \"kubernetes.io/projected/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-kube-api-access-rrh7h\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.877614 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.877565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.877614 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.877591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.877614 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.877610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.978580 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.978546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.978580 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.978588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.978823 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.978612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.978823 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.978628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.978823 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.978678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.978823 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.978704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.978823 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.978764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrh7h\" (UniqueName: \"kubernetes.io/projected/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-kube-api-access-rrh7h\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.978823 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.978802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.978823 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.978824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.979242 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.979058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.979242 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.979187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.979355 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.979318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.979355 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.979326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.979489 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.979471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.980867 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.980839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.981129 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.981111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.988130 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.988107    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:12.988344 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:12.988320    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrh7h\" (UniqueName: \"kubernetes.io/projected/de7a8044-a4f2-47c6-bbdc-04fec9ff8da4-kube-api-access-rrh7h\") pod \"router-gateway-1-openshift-default-54c47cdfc7-wl8wv\" (UID: \"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:13.145787 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:13.145686    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:13.283294 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:13.283267    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"]
Jul 09 18:34:13.285344 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:34:13.285318    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7a8044_a4f2_47c6_bbdc_04fec9ff8da4.slice/crio-11f748d93fc16f69e90b8abbcd5a17bd5a35353c3a8f8d691034e7bf8483521b WatchSource:0}: Error finding container 11f748d93fc16f69e90b8abbcd5a17bd5a35353c3a8f8d691034e7bf8483521b: Status 404 returned error can't find the container with id 11f748d93fc16f69e90b8abbcd5a17bd5a35353c3a8f8d691034e7bf8483521b
Jul 09 18:34:13.287324 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:13.287292    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:34:13.287419 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:13.287370    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:34:13.287419 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:13.287403    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:34:13.593926 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:13.593865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" event={"ID":"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4","Type":"ContainerStarted","Data":"a3f047eea2d4b9769553d4c5676282f6d164c49022ab69481368f20e3c481c38"}
Jul 09 18:34:13.594150 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:13.593945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" event={"ID":"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4","Type":"ContainerStarted","Data":"11f748d93fc16f69e90b8abbcd5a17bd5a35353c3a8f8d691034e7bf8483521b"}
Jul 09 18:34:13.618086 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:13.618030    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" podStartSLOduration=1.618012349 podStartE2EDuration="1.618012349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:13.617072915 +0000 UTC m=+1773.335335678" watchObservedRunningTime="2026-07-09 18:34:13.618012349 +0000 UTC m=+1773.336275112"
Jul 09 18:34:14.146406 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:14.146364    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:14.151271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:14.151246    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:14.598700 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:14.598663    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:14.669036 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:14.669002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:23.646893 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:23.646859    2565 generic.go:356] "Generic (PLEG): container finished" podID="de7a8044-a4f2-47c6-bbdc-04fec9ff8da4" containerID="a3f047eea2d4b9769553d4c5676282f6d164c49022ab69481368f20e3c481c38" exitCode=0
Jul 09 18:34:23.647323 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:23.646921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" event={"ID":"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4","Type":"ContainerDied","Data":"a3f047eea2d4b9769553d4c5676282f6d164c49022ab69481368f20e3c481c38"}
Jul 09 18:34:23.647323 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:23.647275    2565 scope.go:117] "RemoveContainer" containerID="a3f047eea2d4b9769553d4c5676282f6d164c49022ab69481368f20e3c481c38"
Jul 09 18:34:23.648317 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:23.648286    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:34:23.648435 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:23.648350    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:34:23.648435 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:23.648383    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 18:34:24.653920 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.653865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" event={"ID":"de7a8044-a4f2-47c6-bbdc-04fec9ff8da4","Type":"ContainerStarted","Data":"c88dc4b859ceaefb7f43f51cbed5605cfac890243c9d752a17eec785df43129a"}
Jul 09 18:34:24.659682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.659654    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"]
Jul 09 18:34:24.662044 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.662026    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.664791 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.664773    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-n9csv\""
Jul 09 18:34:24.665104 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.665084    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 09 18:34:24.665169 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.665127    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:34:24.675959 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.675922    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"]
Jul 09 18:34:24.783771 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.783719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.784006 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.783850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beefa1cf-6537-46cf-af57-54f6f806f9c6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.784006 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.783920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.784006 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.783950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zg9\" (UniqueName: \"kubernetes.io/projected/beefa1cf-6537-46cf-af57-54f6f806f9c6-kube-api-access-n5zg9\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.885832 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.884672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beefa1cf-6537-46cf-af57-54f6f806f9c6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.885832 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.884748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.885832 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.884791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zg9\" (UniqueName: \"kubernetes.io/projected/beefa1cf-6537-46cf-af57-54f6f806f9c6-kube-api-access-n5zg9\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.885832 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.884834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.885832 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.885337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.886670 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.886645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.893587 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.893548    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beefa1cf-6537-46cf-af57-54f6f806f9c6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.896336 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.896306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zg9\" (UniqueName: \"kubernetes.io/projected/beefa1cf-6537-46cf-af57-54f6f806f9c6-kube-api-access-n5zg9\") pod \"router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:24.971995 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:24.971941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:25.143700 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:25.143675    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"]
Jul 09 18:34:25.145615 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:34:25.145583    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeefa1cf_6537_46cf_af57_54f6f806f9c6.slice/crio-de24513d8a7409a57bc5c5849b8efba9687af12264165a758c5bf0e8322e6427 WatchSource:0}: Error finding container de24513d8a7409a57bc5c5849b8efba9687af12264165a758c5bf0e8322e6427: Status 404 returned error can't find the container with id de24513d8a7409a57bc5c5849b8efba9687af12264165a758c5bf0e8322e6427
Jul 09 18:34:25.146410 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:25.146390    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:25.147393 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:25.147367    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" podUID="de7a8044-a4f2-47c6-bbdc-04fec9ff8da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 09 18:34:25.660652 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:25.660548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" event={"ID":"beefa1cf-6537-46cf-af57-54f6f806f9c6","Type":"ContainerStarted","Data":"2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713"}
Jul 09 18:34:25.660652 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:25.660644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" event={"ID":"beefa1cf-6537-46cf-af57-54f6f806f9c6","Type":"ContainerStarted","Data":"de24513d8a7409a57bc5c5849b8efba9687af12264165a758c5bf0e8322e6427"}
Jul 09 18:34:26.147023 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:26.146977    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" podUID="de7a8044-a4f2-47c6-bbdc-04fec9ff8da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 09 18:34:26.666873 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:26.666834    2565 generic.go:356] "Generic (PLEG): container finished" podID="beefa1cf-6537-46cf-af57-54f6f806f9c6" containerID="2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713" exitCode=0
Jul 09 18:34:26.667285 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:26.666931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" event={"ID":"beefa1cf-6537-46cf-af57-54f6f806f9c6","Type":"ContainerDied","Data":"2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713"}
Jul 09 18:34:27.146148 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:27.146104    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" podUID="de7a8044-a4f2-47c6-bbdc-04fec9ff8da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 09 18:34:27.676926 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:27.676854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" event={"ID":"beefa1cf-6537-46cf-af57-54f6f806f9c6","Type":"ContainerStarted","Data":"02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0"}
Jul 09 18:34:27.678182 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:27.678139    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:34:27.702703 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:27.702649    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" podStartSLOduration=2.926197134 podStartE2EDuration="3.702632852s" totalImagesPullingTime="0s" totalInitContainerRuntime="776.435718ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:27.700787347 +0000 UTC m=+1787.419050112" watchObservedRunningTime="2026-07-09 18:34:27.702632852 +0000 UTC m=+1787.420895616"
Jul 09 18:34:28.147169 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:28.147123    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" podUID="de7a8044-a4f2-47c6-bbdc-04fec9ff8da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 09 18:34:29.146750 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:29.146704    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv" podUID="de7a8044-a4f2-47c6-bbdc-04fec9ff8da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 09 18:34:29.598672 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:29.598635    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:30.150763 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:30.150732    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:30.151741 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:30.151718    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv"
Jul 09 18:34:40.809840 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:40.809808    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:34:40.811561 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:40.811541    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:34:59.691393 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:34:59.691359    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:36:47.035208 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.035167    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"]
Jul 09 18:36:47.046099 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:36:47.046061    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:36:47.046099 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:36:47.046099 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:36:47.046099 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0" cmd=["/bin/sleep","15"]
Jul 09 18:36:47.046389 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.046115    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" message="" err=<
Jul 09 18:36:47.046389 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:36:47.046389 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:36:47.046389 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:36:47.046389 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:36:47.046131    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:36:47.046389 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:36:47.046389 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:36:47.046389 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" podUID="beefa1cf-6537-46cf-af57-54f6f806f9c6" containerName="main" containerID="cri-o://02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0"
Jul 09 18:36:47.046389 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.046177    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" podUID="beefa1cf-6537-46cf-af57-54f6f806f9c6" containerName="main" containerID="cri-o://02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0" gracePeriod=60
Jul 09 18:36:47.309396 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.309366    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:36:47.313456 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.313430    2565 generic.go:356] "Generic (PLEG): container finished" podID="beefa1cf-6537-46cf-af57-54f6f806f9c6" containerID="02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0" exitCode=0
Jul 09 18:36:47.313607 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.313469    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"
Jul 09 18:36:47.313607 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.313518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" event={"ID":"beefa1cf-6537-46cf-af57-54f6f806f9c6","Type":"ContainerDied","Data":"02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0"}
Jul 09 18:36:47.313607 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.313576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s" event={"ID":"beefa1cf-6537-46cf-af57-54f6f806f9c6","Type":"ContainerDied","Data":"de24513d8a7409a57bc5c5849b8efba9687af12264165a758c5bf0e8322e6427"}
Jul 09 18:36:47.313773 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.313608    2565 scope.go:117] "RemoveContainer" containerID="02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0"
Jul 09 18:36:47.322241 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.322224    2565 scope.go:117] "RemoveContainer" containerID="2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713"
Jul 09 18:36:47.330208 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.330187    2565 scope.go:117] "RemoveContainer" containerID="02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0"
Jul 09 18:36:47.330510 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:36:47.330490    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0\": container with ID starting with 02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0 not found: ID does not exist" containerID="02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0"
Jul 09 18:36:47.330563 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.330518    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0"} err="failed to get container status \"02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0\": rpc error: code = NotFound desc = could not find container \"02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0\": container with ID starting with 02cd3d11ae4d83b25deabcdc4ae32f03008b893908c7115c951d827f6cacccc0 not found: ID does not exist"
Jul 09 18:36:47.330563 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.330537    2565 scope.go:117] "RemoveContainer" containerID="2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713"
Jul 09 18:36:47.330787 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:36:47.330774    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713\": container with ID starting with 2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713 not found: ID does not exist" containerID="2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713"
Jul 09 18:36:47.330836 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.330790    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713"} err="failed to get container status \"2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713\": rpc error: code = NotFound desc = could not find container \"2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713\": container with ID starting with 2d728a18c5a85f9c88eb43c0c885143d1a006f6f004175fc89cb52553a378713 not found: ID does not exist"
Jul 09 18:36:47.340082 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.340054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beefa1cf-6537-46cf-af57-54f6f806f9c6-tls-certs\") pod \"beefa1cf-6537-46cf-af57-54f6f806f9c6\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") "
Jul 09 18:36:47.340191 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.340090    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5zg9\" (UniqueName: \"kubernetes.io/projected/beefa1cf-6537-46cf-af57-54f6f806f9c6-kube-api-access-n5zg9\") pod \"beefa1cf-6537-46cf-af57-54f6f806f9c6\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") "
Jul 09 18:36:47.340272 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.340251    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-kserve-provision-location\") pod \"beefa1cf-6537-46cf-af57-54f6f806f9c6\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") "
Jul 09 18:36:47.340411 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.340392    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-tokenizer-uds\") pod \"beefa1cf-6537-46cf-af57-54f6f806f9c6\" (UID: \"beefa1cf-6537-46cf-af57-54f6f806f9c6\") "
Jul 09 18:36:47.340678 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.340654    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "beefa1cf-6537-46cf-af57-54f6f806f9c6" (UID: "beefa1cf-6537-46cf-af57-54f6f806f9c6"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:47.340977 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.340958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "beefa1cf-6537-46cf-af57-54f6f806f9c6" (UID: "beefa1cf-6537-46cf-af57-54f6f806f9c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:47.342570 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.342552    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beefa1cf-6537-46cf-af57-54f6f806f9c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "beefa1cf-6537-46cf-af57-54f6f806f9c6" (UID: "beefa1cf-6537-46cf-af57-54f6f806f9c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:47.342570 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.342558    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beefa1cf-6537-46cf-af57-54f6f806f9c6-kube-api-access-n5zg9" (OuterVolumeSpecName: "kube-api-access-n5zg9") pod "beefa1cf-6537-46cf-af57-54f6f806f9c6" (UID: "beefa1cf-6537-46cf-af57-54f6f806f9c6"). InnerVolumeSpecName "kube-api-access-n5zg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:36:47.442065 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.442027    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beefa1cf-6537-46cf-af57-54f6f806f9c6-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:36:47.442065 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.442062    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5zg9\" (UniqueName: \"kubernetes.io/projected/beefa1cf-6537-46cf-af57-54f6f806f9c6-kube-api-access-n5zg9\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:36:47.442065 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.442072    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:36:47.442307 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.442081    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/beefa1cf-6537-46cf-af57-54f6f806f9c6-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:36:47.643334 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.643297    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"]
Jul 09 18:36:47.657390 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:47.657358    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s"]
Jul 09 18:36:48.868576 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:36:48.868533    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beefa1cf-6537-46cf-af57-54f6f806f9c6" path="/var/lib/kubelet/pods/beefa1cf-6537-46cf-af57-54f6f806f9c6/volumes"
Jul 09 18:37:16.175413 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.175376    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"]
Jul 09 18:37:16.176034 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.175806    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beefa1cf-6537-46cf-af57-54f6f806f9c6" containerName="main"
Jul 09 18:37:16.176034 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.175820    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="beefa1cf-6537-46cf-af57-54f6f806f9c6" containerName="main"
Jul 09 18:37:16.176034 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.175831    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beefa1cf-6537-46cf-af57-54f6f806f9c6" containerName="storage-initializer"
Jul 09 18:37:16.176034 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.175838    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="beefa1cf-6537-46cf-af57-54f6f806f9c6" containerName="storage-initializer"
Jul 09 18:37:16.176034 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.175930    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="beefa1cf-6537-46cf-af57-54f6f806f9c6" containerName="main"
Jul 09 18:37:16.178946 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.178930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.182946 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.182920    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:37:16.183088 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.183019    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 09 18:37:16.183088 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.183079    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-887ld\""
Jul 09 18:37:16.192285 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.192256    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"]
Jul 09 18:37:16.297224 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.297187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvln\" (UniqueName: \"kubernetes.io/projected/91e23e7b-f431-464a-bd5b-6157f05daa2c-kube-api-access-4lvln\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.297403 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.297231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.297403 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.297306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e23e7b-f431-464a-bd5b-6157f05daa2c-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.297403 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.297349    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.398271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.398223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvln\" (UniqueName: \"kubernetes.io/projected/91e23e7b-f431-464a-bd5b-6157f05daa2c-kube-api-access-4lvln\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.398271 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.398273    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.398528 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.398388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e23e7b-f431-464a-bd5b-6157f05daa2c-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.398528 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.398428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.398641 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.398631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.398735 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.398718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.400883 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.400862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e23e7b-f431-464a-bd5b-6157f05daa2c-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.414192 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.414163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvln\" (UniqueName: \"kubernetes.io/projected/91e23e7b-f431-464a-bd5b-6157f05daa2c-kube-api-access-4lvln\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.488962 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.488923    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:16.628951 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:16.628918    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"]
Jul 09 18:37:16.631365 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:37:16.631325    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e23e7b_f431_464a_bd5b_6157f05daa2c.slice/crio-9e96513ddb15bd3c2f9a93fce9163f6fa8346902a81eba5b95df2a7bbca16d19 WatchSource:0}: Error finding container 9e96513ddb15bd3c2f9a93fce9163f6fa8346902a81eba5b95df2a7bbca16d19: Status 404 returned error can't find the container with id 9e96513ddb15bd3c2f9a93fce9163f6fa8346902a81eba5b95df2a7bbca16d19
Jul 09 18:37:17.451339 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:17.451305    2565 generic.go:356] "Generic (PLEG): container finished" podID="91e23e7b-f431-464a-bd5b-6157f05daa2c" containerID="e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2" exitCode=0
Jul 09 18:37:17.451793 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:17.451350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h" event={"ID":"91e23e7b-f431-464a-bd5b-6157f05daa2c","Type":"ContainerDied","Data":"e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2"}
Jul 09 18:37:17.451793 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:17.451395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h" event={"ID":"91e23e7b-f431-464a-bd5b-6157f05daa2c","Type":"ContainerStarted","Data":"9e96513ddb15bd3c2f9a93fce9163f6fa8346902a81eba5b95df2a7bbca16d19"}
Jul 09 18:37:18.458029 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:18.457982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h" event={"ID":"91e23e7b-f431-464a-bd5b-6157f05daa2c","Type":"ContainerStarted","Data":"79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49"}
Jul 09 18:37:18.458430 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:18.458066    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:37:18.492764 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:18.492713    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h" podStartSLOduration=1.8579815119999998 podStartE2EDuration="2.49269698s" totalImagesPullingTime="0s" totalInitContainerRuntime="634.715468ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:18.488455405 +0000 UTC m=+1958.206718169" watchObservedRunningTime="2026-07-09 18:37:18.49269698 +0000 UTC m=+1958.210959744"
Jul 09 18:37:49.464249 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:37:49.464218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:39:40.836696 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:39:40.836665    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:39:40.839966 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:39:40.839944    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:40:31.701190 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:31.701094    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"]
Jul 09 18:40:31.710035 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:40:31.709997    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:40:31.710035 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:40:31.710035 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:40:31.710035 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49" cmd=["/bin/sleep","15"]
Jul 09 18:40:31.710328 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:31.710053    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h" message="" err=<
Jul 09 18:40:31.710328 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:40:31.710328 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:40:31.710328 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:40:31.710328 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:40:31.710068    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:40:31.710328 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:40:31.710328 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:40:31.710328 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h" podUID="91e23e7b-f431-464a-bd5b-6157f05daa2c" containerName="main" containerID="cri-o://79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49"
Jul 09 18:40:31.710328 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:31.710123    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h" podUID="91e23e7b-f431-464a-bd5b-6157f05daa2c" containerName="main" containerID="cri-o://79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49" gracePeriod=60
Jul 09 18:40:31.952736 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:31.952677    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:40:32.111967 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.111893    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-kserve-provision-location\") pod \"91e23e7b-f431-464a-bd5b-6157f05daa2c\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") "
Jul 09 18:40:32.112150 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.112058    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-tokenizer-uds\") pod \"91e23e7b-f431-464a-bd5b-6157f05daa2c\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") "
Jul 09 18:40:32.112150 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.112089    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e23e7b-f431-464a-bd5b-6157f05daa2c-tls-certs\") pod \"91e23e7b-f431-464a-bd5b-6157f05daa2c\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") "
Jul 09 18:40:32.112150 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.112134    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lvln\" (UniqueName: \"kubernetes.io/projected/91e23e7b-f431-464a-bd5b-6157f05daa2c-kube-api-access-4lvln\") pod \"91e23e7b-f431-464a-bd5b-6157f05daa2c\" (UID: \"91e23e7b-f431-464a-bd5b-6157f05daa2c\") "
Jul 09 18:40:32.112340 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.112314    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "91e23e7b-f431-464a-bd5b-6157f05daa2c" (UID: "91e23e7b-f431-464a-bd5b-6157f05daa2c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:32.112406 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.112388    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:40:32.112672 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.112648    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91e23e7b-f431-464a-bd5b-6157f05daa2c" (UID: "91e23e7b-f431-464a-bd5b-6157f05daa2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:32.114222 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.114192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e23e7b-f431-464a-bd5b-6157f05daa2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91e23e7b-f431-464a-bd5b-6157f05daa2c" (UID: "91e23e7b-f431-464a-bd5b-6157f05daa2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:40:32.114338 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.114254    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e23e7b-f431-464a-bd5b-6157f05daa2c-kube-api-access-4lvln" (OuterVolumeSpecName: "kube-api-access-4lvln") pod "91e23e7b-f431-464a-bd5b-6157f05daa2c" (UID: "91e23e7b-f431-464a-bd5b-6157f05daa2c"). InnerVolumeSpecName "kube-api-access-4lvln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:40:32.213339 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.213251    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e23e7b-f431-464a-bd5b-6157f05daa2c-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:40:32.213339 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.213283    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lvln\" (UniqueName: \"kubernetes.io/projected/91e23e7b-f431-464a-bd5b-6157f05daa2c-kube-api-access-4lvln\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:40:32.213339 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.213294    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91e23e7b-f431-464a-bd5b-6157f05daa2c-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:40:32.321241 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.321205    2565 generic.go:356] "Generic (PLEG): container finished" podID="91e23e7b-f431-464a-bd5b-6157f05daa2c" containerID="79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49" exitCode=0
Jul 09 18:40:32.321439 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.321257    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"
Jul 09 18:40:32.321439 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.321289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h" event={"ID":"91e23e7b-f431-464a-bd5b-6157f05daa2c","Type":"ContainerDied","Data":"79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49"}
Jul 09 18:40:32.321439 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.321348    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h" event={"ID":"91e23e7b-f431-464a-bd5b-6157f05daa2c","Type":"ContainerDied","Data":"9e96513ddb15bd3c2f9a93fce9163f6fa8346902a81eba5b95df2a7bbca16d19"}
Jul 09 18:40:32.321439 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.321379    2565 scope.go:117] "RemoveContainer" containerID="79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49"
Jul 09 18:40:32.330733 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.330717    2565 scope.go:117] "RemoveContainer" containerID="e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2"
Jul 09 18:40:32.339842 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.339821    2565 scope.go:117] "RemoveContainer" containerID="79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49"
Jul 09 18:40:32.340088 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:40:32.340070    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49\": container with ID starting with 79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49 not found: ID does not exist" containerID="79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49"
Jul 09 18:40:32.340144 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.340095    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49"} err="failed to get container status \"79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49\": rpc error: code = NotFound desc = could not find container \"79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49\": container with ID starting with 79833606110ffb41b10c8fdae69c54bf493f4cc4424fde0c5085485767c18b49 not found: ID does not exist"
Jul 09 18:40:32.340144 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.340114    2565 scope.go:117] "RemoveContainer" containerID="e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2"
Jul 09 18:40:32.340376 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:40:32.340359    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2\": container with ID starting with e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2 not found: ID does not exist" containerID="e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2"
Jul 09 18:40:32.340429 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.340381    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2"} err="failed to get container status \"e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2\": rpc error: code = NotFound desc = could not find container \"e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2\": container with ID starting with e92710fa0888d3c6a7a4e9e9a8b620d5de4baa2962b4286a33e798f8943cbfd2 not found: ID does not exist"
Jul 09 18:40:32.344633 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.344610    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"]
Jul 09 18:40:32.348889 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.348866    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h"]
Jul 09 18:40:32.869685 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:32.869651    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e23e7b-f431-464a-bd5b-6157f05daa2c" path="/var/lib/kubelet/pods/91e23e7b-f431-464a-bd5b-6157f05daa2c/volumes"
Jul 09 18:40:59.495718 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.495687    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"]
Jul 09 18:40:59.496139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.496029    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e23e7b-f431-464a-bd5b-6157f05daa2c" containerName="main"
Jul 09 18:40:59.496139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.496041    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e23e7b-f431-464a-bd5b-6157f05daa2c" containerName="main"
Jul 09 18:40:59.496139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.496050    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e23e7b-f431-464a-bd5b-6157f05daa2c" containerName="storage-initializer"
Jul 09 18:40:59.496139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.496057    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e23e7b-f431-464a-bd5b-6157f05daa2c" containerName="storage-initializer"
Jul 09 18:40:59.496139 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.496112    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e23e7b-f431-464a-bd5b-6157f05daa2c" containerName="main"
Jul 09 18:40:59.500619 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.500598    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.503566 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.503543    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:40:59.503708 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.503543    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-l6d5n\""
Jul 09 18:40:59.503708 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.503676    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 09 18:40:59.510406 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.510381    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"]
Jul 09 18:40:59.540245 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.540214    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.540245 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.540247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5489e6-c90a-4987-ad9b-19f52c97e380-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.540431 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.540342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhmk\" (UniqueName: \"kubernetes.io/projected/8e5489e6-c90a-4987-ad9b-19f52c97e380-kube-api-access-xjhmk\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.540431 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.540385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.641082 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.641039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhmk\" (UniqueName: \"kubernetes.io/projected/8e5489e6-c90a-4987-ad9b-19f52c97e380-kube-api-access-xjhmk\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.641267 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.641094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.641267 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.641131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.641267 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.641150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5489e6-c90a-4987-ad9b-19f52c97e380-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.641499 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.641477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.641539 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.641525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.643676 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.643653    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5489e6-c90a-4987-ad9b-19f52c97e380-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.655371 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.655334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhmk\" (UniqueName: \"kubernetes.io/projected/8e5489e6-c90a-4987-ad9b-19f52c97e380-kube-api-access-xjhmk\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.810522 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.810421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:40:59.947526 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:40:59.947498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"]
Jul 09 18:40:59.949946 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:40:59.949899    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5489e6_c90a_4987_ad9b_19f52c97e380.slice/crio-aa4e9b6dcd7a28246da0fd83b7bebb026a7cb88ee12fd5fd411b3854673c0266 WatchSource:0}: Error finding container aa4e9b6dcd7a28246da0fd83b7bebb026a7cb88ee12fd5fd411b3854673c0266: Status 404 returned error can't find the container with id aa4e9b6dcd7a28246da0fd83b7bebb026a7cb88ee12fd5fd411b3854673c0266
Jul 09 18:41:00.444865 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:41:00.444826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" event={"ID":"8e5489e6-c90a-4987-ad9b-19f52c97e380","Type":"ContainerStarted","Data":"14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e"}
Jul 09 18:41:00.445030 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:41:00.444879    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" event={"ID":"8e5489e6-c90a-4987-ad9b-19f52c97e380","Type":"ContainerStarted","Data":"aa4e9b6dcd7a28246da0fd83b7bebb026a7cb88ee12fd5fd411b3854673c0266"}
Jul 09 18:41:01.450098 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:41:01.450067    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e5489e6-c90a-4987-ad9b-19f52c97e380" containerID="14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e" exitCode=0
Jul 09 18:41:01.450545 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:41:01.450155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" event={"ID":"8e5489e6-c90a-4987-ad9b-19f52c97e380","Type":"ContainerDied","Data":"14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e"}
Jul 09 18:41:02.456489 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:41:02.456448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" event={"ID":"8e5489e6-c90a-4987-ad9b-19f52c97e380","Type":"ContainerStarted","Data":"ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be"}
Jul 09 18:41:02.456887 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:41:02.456545    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:41:02.479036 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:41:02.478984    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" podStartSLOduration=2.766766853 podStartE2EDuration="3.478967652s" totalImagesPullingTime="0s" totalInitContainerRuntime="712.200799ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:41:02.476748486 +0000 UTC m=+2182.195011261" watchObservedRunningTime="2026-07-09 18:41:02.478967652 +0000 UTC m=+2182.197230419"
Jul 09 18:41:33.463403 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:41:33.463374    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:44:26.541682 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.541641    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"]
Jul 09 18:44:26.550967 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:44:26.550898    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:44:26.550967 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:44:26.550967 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:44:26.550967 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be" cmd=["/bin/sleep","15"]
Jul 09 18:44:26.551219 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.550978    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" message="" err=<
Jul 09 18:44:26.551219 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:44:26.551219 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:44:26.551219 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:44:26.551219 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:44:26.550991    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:44:26.551219 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:44:26.551219 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:44:26.551219 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" podUID="8e5489e6-c90a-4987-ad9b-19f52c97e380" containerName="main" containerID="cri-o://ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be"
Jul 09 18:44:26.551219 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.551038    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" podUID="8e5489e6-c90a-4987-ad9b-19f52c97e380" containerName="main" containerID="cri-o://ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be" gracePeriod=60
Jul 09 18:44:26.805484 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.805414    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:44:26.841238 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.841195    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjhmk\" (UniqueName: \"kubernetes.io/projected/8e5489e6-c90a-4987-ad9b-19f52c97e380-kube-api-access-xjhmk\") pod \"8e5489e6-c90a-4987-ad9b-19f52c97e380\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") "
Jul 09 18:44:26.841427 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.841247    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5489e6-c90a-4987-ad9b-19f52c97e380-tls-certs\") pod \"8e5489e6-c90a-4987-ad9b-19f52c97e380\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") "
Jul 09 18:44:26.841427 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.841355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-kserve-provision-location\") pod \"8e5489e6-c90a-4987-ad9b-19f52c97e380\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") "
Jul 09 18:44:26.841427 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.841379    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-tokenizer-uds\") pod \"8e5489e6-c90a-4987-ad9b-19f52c97e380\" (UID: \"8e5489e6-c90a-4987-ad9b-19f52c97e380\") "
Jul 09 18:44:26.841738 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.841706    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8e5489e6-c90a-4987-ad9b-19f52c97e380" (UID: "8e5489e6-c90a-4987-ad9b-19f52c97e380"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:26.842087 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.842065    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e5489e6-c90a-4987-ad9b-19f52c97e380" (UID: "8e5489e6-c90a-4987-ad9b-19f52c97e380"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:26.843442 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.843414    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5489e6-c90a-4987-ad9b-19f52c97e380-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e5489e6-c90a-4987-ad9b-19f52c97e380" (UID: "8e5489e6-c90a-4987-ad9b-19f52c97e380"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:44:26.843527 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.843440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5489e6-c90a-4987-ad9b-19f52c97e380-kube-api-access-xjhmk" (OuterVolumeSpecName: "kube-api-access-xjhmk") pod "8e5489e6-c90a-4987-ad9b-19f52c97e380" (UID: "8e5489e6-c90a-4987-ad9b-19f52c97e380"). InnerVolumeSpecName "kube-api-access-xjhmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:44:26.942430 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.942387    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:44:26.942430 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.942418    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e5489e6-c90a-4987-ad9b-19f52c97e380-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:44:26.942430 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.942428    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjhmk\" (UniqueName: \"kubernetes.io/projected/8e5489e6-c90a-4987-ad9b-19f52c97e380-kube-api-access-xjhmk\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:44:26.942430 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:26.942437    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e5489e6-c90a-4987-ad9b-19f52c97e380-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:44:27.393248 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.393216    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e5489e6-c90a-4987-ad9b-19f52c97e380" containerID="ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be" exitCode=0
Jul 09 18:44:27.393457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.393267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" event={"ID":"8e5489e6-c90a-4987-ad9b-19f52c97e380","Type":"ContainerDied","Data":"ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be"}
Jul 09 18:44:27.393457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.393304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg" event={"ID":"8e5489e6-c90a-4987-ad9b-19f52c97e380","Type":"ContainerDied","Data":"aa4e9b6dcd7a28246da0fd83b7bebb026a7cb88ee12fd5fd411b3854673c0266"}
Jul 09 18:44:27.393457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.393323    2565 scope.go:117] "RemoveContainer" containerID="ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be"
Jul 09 18:44:27.393457 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.393267    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"
Jul 09 18:44:27.404834 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.404805    2565 scope.go:117] "RemoveContainer" containerID="14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e"
Jul 09 18:44:27.413159 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.413134    2565 scope.go:117] "RemoveContainer" containerID="ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be"
Jul 09 18:44:27.413476 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:44:27.413443    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be\": container with ID starting with ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be not found: ID does not exist" containerID="ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be"
Jul 09 18:44:27.413625 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.413487    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be"} err="failed to get container status \"ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be\": rpc error: code = NotFound desc = could not find container \"ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be\": container with ID starting with ca1ad15d1edf0807bb4b4f167df3b8bc3b2c671c2ef9a291fa9994458a3186be not found: ID does not exist"
Jul 09 18:44:27.413625 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.413514    2565 scope.go:117] "RemoveContainer" containerID="14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e"
Jul 09 18:44:27.414352 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:44:27.414212    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e\": container with ID starting with 14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e not found: ID does not exist" containerID="14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e"
Jul 09 18:44:27.414352 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.414246    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e"} err="failed to get container status \"14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e\": rpc error: code = NotFound desc = could not find container \"14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e\": container with ID starting with 14895d8c38f69ca186e585fc22948aa0a62c873246d1350870c3e49e8ad3003e not found: ID does not exist"
Jul 09 18:44:27.415993 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.415969    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"]
Jul 09 18:44:27.423108 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:27.423083    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg"]
Jul 09 18:44:28.868726 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:28.868691    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5489e6-c90a-4987-ad9b-19f52c97e380" path="/var/lib/kubelet/pods/8e5489e6-c90a-4987-ad9b-19f52c97e380/volumes"
Jul 09 18:44:40.864274 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:40.864150    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:44:40.868462 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:44:40.868439    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:45:13.651673 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.651639    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"]
Jul 09 18:45:13.652205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.651968    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5489e6-c90a-4987-ad9b-19f52c97e380" containerName="main"
Jul 09 18:45:13.652205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.651980    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5489e6-c90a-4987-ad9b-19f52c97e380" containerName="main"
Jul 09 18:45:13.652205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.652003    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5489e6-c90a-4987-ad9b-19f52c97e380" containerName="storage-initializer"
Jul 09 18:45:13.652205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.652009    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5489e6-c90a-4987-ad9b-19f52c97e380" containerName="storage-initializer"
Jul 09 18:45:13.652205 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.652065    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5489e6-c90a-4987-ad9b-19f52c97e380" containerName="main"
Jul 09 18:45:13.656261 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.656241    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.659055 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.659031    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:45:13.659124 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.659072    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 09 18:45:13.659124 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.659088    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-z5wrf\""
Jul 09 18:45:13.666687 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.666663    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"]
Jul 09 18:45:13.756851 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.756816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.757066 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.756859    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1556e243-3382-4bd0-9af4-cebbfb78db6b-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.757066 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.756881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.757066 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.756898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptpfl\" (UniqueName: \"kubernetes.io/projected/1556e243-3382-4bd0-9af4-cebbfb78db6b-kube-api-access-ptpfl\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.858300 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.858263    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.858300 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.858308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1556e243-3382-4bd0-9af4-cebbfb78db6b-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.858567 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.858326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.858567 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.858347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptpfl\" (UniqueName: \"kubernetes.io/projected/1556e243-3382-4bd0-9af4-cebbfb78db6b-kube-api-access-ptpfl\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.858744 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.858722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.858808 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.858732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.860810 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.860786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1556e243-3382-4bd0-9af4-cebbfb78db6b-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.867355 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.867333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptpfl\" (UniqueName: \"kubernetes.io/projected/1556e243-3382-4bd0-9af4-cebbfb78db6b-kube-api-access-ptpfl\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:13.966572 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:13.966532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:14.102561 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:14.102530    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"]
Jul 09 18:45:14.104523 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:45:14.104488    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1556e243_3382_4bd0_9af4_cebbfb78db6b.slice/crio-2c8838a9c2bfed6c6df99e8854e2f04b2a2f0e2f4f26d6de77dbd4152091722f WatchSource:0}: Error finding container 2c8838a9c2bfed6c6df99e8854e2f04b2a2f0e2f4f26d6de77dbd4152091722f: Status 404 returned error can't find the container with id 2c8838a9c2bfed6c6df99e8854e2f04b2a2f0e2f4f26d6de77dbd4152091722f
Jul 09 18:45:14.601965 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:14.601933    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" event={"ID":"1556e243-3382-4bd0-9af4-cebbfb78db6b","Type":"ContainerStarted","Data":"5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9"}
Jul 09 18:45:14.602169 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:14.601983    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" event={"ID":"1556e243-3382-4bd0-9af4-cebbfb78db6b","Type":"ContainerStarted","Data":"2c8838a9c2bfed6c6df99e8854e2f04b2a2f0e2f4f26d6de77dbd4152091722f"}
Jul 09 18:45:15.608464 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:15.608429    2565 generic.go:356] "Generic (PLEG): container finished" podID="1556e243-3382-4bd0-9af4-cebbfb78db6b" containerID="5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9" exitCode=0
Jul 09 18:45:15.608887 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:15.608519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" event={"ID":"1556e243-3382-4bd0-9af4-cebbfb78db6b","Type":"ContainerDied","Data":"5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9"}
Jul 09 18:45:16.615143 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:16.615099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" event={"ID":"1556e243-3382-4bd0-9af4-cebbfb78db6b","Type":"ContainerStarted","Data":"1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6"}
Jul 09 18:45:16.615606 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:16.615176    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:45:16.637317 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:16.637263    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" podStartSLOduration=2.980636706 podStartE2EDuration="3.637246494s" totalImagesPullingTime="0s" totalInitContainerRuntime="656.609788ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:45:16.63582302 +0000 UTC m=+2436.354085787" watchObservedRunningTime="2026-07-09 18:45:16.637246494 +0000 UTC m=+2436.355509259"
Jul 09 18:45:47.621816 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:45:47.621779    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:46:33.121159 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:46:33.121079    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 18:49:03.240687 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.239708    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"]
Jul 09 18:49:03.249386 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:49:03.249328    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:49:03.249386 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:49:03.249386 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:49:03.249386 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6" cmd=["/bin/sleep","15"]
Jul 09 18:49:03.249715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.249397    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" message="" err=<
Jul 09 18:49:03.249715 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:49:03.249715 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:49:03.249715 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 18:49:03.249715 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:49:03.249409    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:49:03.249715 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:49:03.249715 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 18:49:03.249715 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" podUID="1556e243-3382-4bd0-9af4-cebbfb78db6b" containerName="main" containerID="cri-o://1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6"
Jul 09 18:49:03.249715 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.249458    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" podUID="1556e243-3382-4bd0-9af4-cebbfb78db6b" containerName="main" containerID="cri-o://1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6" gracePeriod=60
Jul 09 18:49:03.496166 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.496102    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:49:03.588410 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.588368    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-kserve-provision-location\") pod \"1556e243-3382-4bd0-9af4-cebbfb78db6b\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") "
Jul 09 18:49:03.588598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.588462    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptpfl\" (UniqueName: \"kubernetes.io/projected/1556e243-3382-4bd0-9af4-cebbfb78db6b-kube-api-access-ptpfl\") pod \"1556e243-3382-4bd0-9af4-cebbfb78db6b\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") "
Jul 09 18:49:03.588598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.588485    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-tokenizer-uds\") pod \"1556e243-3382-4bd0-9af4-cebbfb78db6b\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") "
Jul 09 18:49:03.588598 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.588533    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1556e243-3382-4bd0-9af4-cebbfb78db6b-tls-certs\") pod \"1556e243-3382-4bd0-9af4-cebbfb78db6b\" (UID: \"1556e243-3382-4bd0-9af4-cebbfb78db6b\") "
Jul 09 18:49:03.588765 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.588744    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1556e243-3382-4bd0-9af4-cebbfb78db6b" (UID: "1556e243-3382-4bd0-9af4-cebbfb78db6b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:03.589095 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.589070    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1556e243-3382-4bd0-9af4-cebbfb78db6b" (UID: "1556e243-3382-4bd0-9af4-cebbfb78db6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:49:03.590641 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.590619    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1556e243-3382-4bd0-9af4-cebbfb78db6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1556e243-3382-4bd0-9af4-cebbfb78db6b" (UID: "1556e243-3382-4bd0-9af4-cebbfb78db6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:49:03.590714 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.590671    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1556e243-3382-4bd0-9af4-cebbfb78db6b-kube-api-access-ptpfl" (OuterVolumeSpecName: "kube-api-access-ptpfl") pod "1556e243-3382-4bd0-9af4-cebbfb78db6b" (UID: "1556e243-3382-4bd0-9af4-cebbfb78db6b"). InnerVolumeSpecName "kube-api-access-ptpfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:49:03.659049 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.659015    2565 generic.go:356] "Generic (PLEG): container finished" podID="1556e243-3382-4bd0-9af4-cebbfb78db6b" containerID="1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6" exitCode=0
Jul 09 18:49:03.659218 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.659066    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"
Jul 09 18:49:03.659218 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.659099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" event={"ID":"1556e243-3382-4bd0-9af4-cebbfb78db6b","Type":"ContainerDied","Data":"1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6"}
Jul 09 18:49:03.659218 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.659153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft" event={"ID":"1556e243-3382-4bd0-9af4-cebbfb78db6b","Type":"ContainerDied","Data":"2c8838a9c2bfed6c6df99e8854e2f04b2a2f0e2f4f26d6de77dbd4152091722f"}
Jul 09 18:49:03.659218 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.659179    2565 scope.go:117] "RemoveContainer" containerID="1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6"
Jul 09 18:49:03.668201 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.668183    2565 scope.go:117] "RemoveContainer" containerID="5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9"
Jul 09 18:49:03.676180 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.676158    2565 scope.go:117] "RemoveContainer" containerID="1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6"
Jul 09 18:49:03.676490 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:49:03.676467    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6\": container with ID starting with 1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6 not found: ID does not exist" containerID="1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6"
Jul 09 18:49:03.676542 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.676500    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6"} err="failed to get container status \"1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6\": rpc error: code = NotFound desc = could not find container \"1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6\": container with ID starting with 1c2b42a212e7380907c57a009e57f4bd2853199ff461cc0ca202b88ca9a6d7d6 not found: ID does not exist"
Jul 09 18:49:03.676542 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.676518    2565 scope.go:117] "RemoveContainer" containerID="5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9"
Jul 09 18:49:03.676787 ip-10-0-129-87 kubenswrapper[2565]: E0709 18:49:03.676764    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9\": container with ID starting with 5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9 not found: ID does not exist" containerID="5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9"
Jul 09 18:49:03.676828 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.676793    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9"} err="failed to get container status \"5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9\": rpc error: code = NotFound desc = could not find container \"5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9\": container with ID starting with 5feb8af74b683b94b13a34a27863d7e6894481c3efa4babf3e57e367f29f79d9 not found: ID does not exist"
Jul 09 18:49:03.684847 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.684817    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"]
Jul 09 18:49:03.689843 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.689816    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:49:03.689843 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.689843    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptpfl\" (UniqueName: \"kubernetes.io/projected/1556e243-3382-4bd0-9af4-cebbfb78db6b-kube-api-access-ptpfl\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:49:03.690022 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.689856    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1556e243-3382-4bd0-9af4-cebbfb78db6b-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:49:03.690022 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.689865    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1556e243-3382-4bd0-9af4-cebbfb78db6b-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 18:49:03.691387 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:03.691369    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft"]
Jul 09 18:49:04.868295 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:04.868266    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1556e243-3382-4bd0-9af4-cebbfb78db6b" path="/var/lib/kubelet/pods/1556e243-3382-4bd0-9af4-cebbfb78db6b/volumes"
Jul 09 18:49:40.891524 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:40.891415    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:49:40.895946 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:49:40.895925    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:53:34.438006 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.437965    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 18:53:34.438427 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.438286    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1556e243-3382-4bd0-9af4-cebbfb78db6b" containerName="main"
Jul 09 18:53:34.438427 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.438296    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1556e243-3382-4bd0-9af4-cebbfb78db6b" containerName="main"
Jul 09 18:53:34.438427 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.438320    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1556e243-3382-4bd0-9af4-cebbfb78db6b" containerName="storage-initializer"
Jul 09 18:53:34.438427 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.438326    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1556e243-3382-4bd0-9af4-cebbfb78db6b" containerName="storage-initializer"
Jul 09 18:53:34.438427 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.438372    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1556e243-3382-4bd0-9af4-cebbfb78db6b" containerName="main"
Jul 09 18:53:34.441471 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.441450    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.444142 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.444118    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-dr75p\""
Jul 09 18:53:34.444275 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.444257    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:53:34.445238 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.445219    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 09 18:53:34.457431 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.457405    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 18:53:34.482073 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.482045    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"]
Jul 09 18:53:34.485297 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.485265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg846\" (UniqueName: \"kubernetes.io/projected/b20149cf-88f3-4d0a-b7a4-3862df357e87-kube-api-access-rg846\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.485429 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.485381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.485497 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.485433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.485497 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.485486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.485605 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.485510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.485605 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.485537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.485705 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.485648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b20149cf-88f3-4d0a-b7a4-3862df357e87-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.485781 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.485756    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.489172 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.489150    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-n998q\""
Jul 09 18:53:34.496358 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.496337    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"]
Jul 09 18:53:34.587008 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.586973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587190 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.587190 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587190 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587190 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587190 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587190 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj87q\" (UniqueName: \"kubernetes.io/projected/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kube-api-access-vj87q\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.587190 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b20149cf-88f3-4d0a-b7a4-3862df357e87-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587535 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.587535 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587535 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg846\" (UniqueName: \"kubernetes.io/projected/b20149cf-88f3-4d0a-b7a4-3862df357e87-kube-api-access-rg846\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587535 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.587535 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587507    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587535 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.587796 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.587599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.589222 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.589199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.589461 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.589442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b20149cf-88f3-4d0a-b7a4-3862df357e87-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.601996 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.601969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg846\" (UniqueName: \"kubernetes.io/projected/b20149cf-88f3-4d0a-b7a4-3862df357e87-kube-api-access-rg846\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.688975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.688868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj87q\" (UniqueName: \"kubernetes.io/projected/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kube-api-access-vj87q\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.688975 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.688936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.689169 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.688992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.689169 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.689045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.689385 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.689363    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.689466 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.689447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.691433 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.691414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.700093 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.700064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj87q\" (UniqueName: \"kubernetes.io/projected/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kube-api-access-vj87q\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.751496 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.751463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:53:34.794777 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.794730    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:34.899850 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.899804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 18:53:34.901886 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:53:34.901836    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20149cf_88f3_4d0a_b7a4_3862df357e87.slice/crio-f24e06b5427bcf683624972bb440c054aec65d0f45f278d07b4c615850934287 WatchSource:0}: Error finding container f24e06b5427bcf683624972bb440c054aec65d0f45f278d07b4c615850934287: Status 404 returned error can't find the container with id f24e06b5427bcf683624972bb440c054aec65d0f45f278d07b4c615850934287
Jul 09 18:53:34.946100 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:34.946074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"]
Jul 09 18:53:34.953047 ip-10-0-129-87 kubenswrapper[2565]: W0709 18:53:34.953019    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510ffc95_67a0_43dd_a57c_55bf22ccaa20.slice/crio-3450848066aa68a03cc70bf0d523fbb77cb33fcf91e3509b474da834d82f091b WatchSource:0}: Error finding container 3450848066aa68a03cc70bf0d523fbb77cb33fcf91e3509b474da834d82f091b: Status 404 returned error can't find the container with id 3450848066aa68a03cc70bf0d523fbb77cb33fcf91e3509b474da834d82f091b
Jul 09 18:53:35.885121 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:35.885063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b20149cf-88f3-4d0a-b7a4-3862df357e87","Type":"ContainerStarted","Data":"0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb"}
Jul 09 18:53:35.885696 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:35.885167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b20149cf-88f3-4d0a-b7a4-3862df357e87","Type":"ContainerStarted","Data":"f24e06b5427bcf683624972bb440c054aec65d0f45f278d07b4c615850934287"}
Jul 09 18:53:35.887600 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:35.887575    2565 generic.go:356] "Generic (PLEG): container finished" podID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" containerID="f67819274d05f5ff9e0b456580be524b9fb441d489c463fc31e68c2ca1c758c9" exitCode=0
Jul 09 18:53:35.887838 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:35.887633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw" event={"ID":"510ffc95-67a0-43dd-a57c-55bf22ccaa20","Type":"ContainerDied","Data":"f67819274d05f5ff9e0b456580be524b9fb441d489c463fc31e68c2ca1c758c9"}
Jul 09 18:53:35.887838 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:35.887666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw" event={"ID":"510ffc95-67a0-43dd-a57c-55bf22ccaa20","Type":"ContainerStarted","Data":"3450848066aa68a03cc70bf0d523fbb77cb33fcf91e3509b474da834d82f091b"}
Jul 09 18:53:36.894138 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:36.894088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw" event={"ID":"510ffc95-67a0-43dd-a57c-55bf22ccaa20","Type":"ContainerStarted","Data":"769c6f6763fb9f13b50d3b20e4e243095466b88fdda0ed53065e6d3a07bb1654"}
Jul 09 18:53:36.894516 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:36.894354    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:53:36.917006 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:36.916958    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw" podStartSLOduration=2.24636272 podStartE2EDuration="2.916941963s" totalImagesPullingTime="0s" totalInitContainerRuntime="670.579243ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:36.915895729 +0000 UTC m=+2936.634158493" watchObservedRunningTime="2026-07-09 18:53:36.916941963 +0000 UTC m=+2936.635204724"
Jul 09 18:53:38.906986 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:38.906949    2565 generic.go:356] "Generic (PLEG): container finished" podID="b20149cf-88f3-4d0a-b7a4-3862df357e87" containerID="0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb" exitCode=0
Jul 09 18:53:38.907437 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:38.907013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b20149cf-88f3-4d0a-b7a4-3862df357e87","Type":"ContainerDied","Data":"0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb"}
Jul 09 18:53:38.908401 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:53:38.908382    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:54:07.903138 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:54:07.903046    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 18:54:23.143819 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:54:23.143780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b20149cf-88f3-4d0a-b7a4-3862df357e87","Type":"ContainerStarted","Data":"3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399"}
Jul 09 18:54:23.168079 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:54:23.168026    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.053886595 podStartE2EDuration="49.168006833s" totalImagesPullingTime="43.375666704s" totalInitContainerRuntime="3.738453534s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:23.166311139 +0000 UTC m=+2982.884573903" watchObservedRunningTime="2026-07-09 18:54:23.168006833 +0000 UTC m=+2982.886269605"
Jul 09 18:54:40.921089 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:54:40.920979    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:54:40.978083 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:54:40.925563    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:59:40.949191 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:59:40.949163    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 18:59:40.955422 ip-10-0-129-87 kubenswrapper[2565]: I0709 18:59:40.955394    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:02:05.757018 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.756975    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 19:02:05.760451 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.760434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.765326 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.765305    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 09 19:02:05.765326 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.765318    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-k5ff9\""
Jul 09 19:02:05.796185 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.795046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.796185 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.795126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.796185 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.795206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.796185 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.795276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7942d565-ff47-4f37-9860-c62d39548388-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.796185 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.795359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.796882 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.796853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 19:02:05.899759 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.899718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.899996 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.899777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7942d565-ff47-4f37-9860-c62d39548388-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.899996 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.899807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.899996 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.899874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.899996 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.899941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjcb\" (UniqueName: \"kubernetes.io/projected/7942d565-ff47-4f37-9860-c62d39548388-kube-api-access-gjjcb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.899996 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.899976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.900216 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.900155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.900265 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.900240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.900322 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.900279    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.902020 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.901994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:05.902282 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:05.902262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7942d565-ff47-4f37-9860-c62d39548388-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:06.001306 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:06.001268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjcb\" (UniqueName: \"kubernetes.io/projected/7942d565-ff47-4f37-9860-c62d39548388-kube-api-access-gjjcb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:06.022242 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:06.022170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjcb\" (UniqueName: \"kubernetes.io/projected/7942d565-ff47-4f37-9860-c62d39548388-kube-api-access-gjjcb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:11.257199 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:11.257166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:11.266258 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:11.266230    2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:11.358400 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:11.358360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:11.358577 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:11.358508    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:11.362502 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:11.362458    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") DevicePath \"csi-55bd14af576e77d6436311c1b9c2889f6970de0650f144b747e86982125a0cd9\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:11.498003 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:11.497975    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/5194e7d566e20e82448dd728d8bd83db3a63ffe4ee741a7ad074e0f3f76777f7/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:11.510717 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:11.510647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:11.770731 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:11.770633    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:11.910106 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:11.910078    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 19:02:11.911700 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:02:11.911675    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7942d565_ff47_4f37_9860_c62d39548388.slice/crio-f3c0cb1ed42f2c182ee8ecaf2f04ec4c345eb59b66a7f89a824d25a3d9dde93f WatchSource:0}: Error finding container f3c0cb1ed42f2c182ee8ecaf2f04ec4c345eb59b66a7f89a824d25a3d9dde93f: Status 404 returned error can't find the container with id f3c0cb1ed42f2c182ee8ecaf2f04ec4c345eb59b66a7f89a824d25a3d9dde93f
Jul 09 19:02:12.383280 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:12.383245    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"7942d565-ff47-4f37-9860-c62d39548388","Type":"ContainerStarted","Data":"c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1"}
Jul 09 19:02:12.383716 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:12.383303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"7942d565-ff47-4f37-9860-c62d39548388","Type":"ContainerStarted","Data":"f3c0cb1ed42f2c182ee8ecaf2f04ec4c345eb59b66a7f89a824d25a3d9dde93f"}
Jul 09 19:02:12.406735 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:12.406665    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=7.406643292 podStartE2EDuration="7.406643292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:12.405854749 +0000 UTC m=+3452.124117513" watchObservedRunningTime="2026-07-09 19:02:12.406643292 +0000 UTC m=+3452.124906057"
Jul 09 19:02:12.770848 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:12.770807    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:21.771371 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:21.771334    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:02:21.772917 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:21.772870    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:02:31.771074 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:31.771027    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:02:41.771974 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:41.771926    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:02:51.771412 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:02:51.771361    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:03:01.771549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:01.771436    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:03:11.771645 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:11.771604    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:03:21.772015 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:21.771974    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:03:25.531146 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.531112    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"]
Jul 09 19:03:25.541669 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:03:25.541628    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:03:25.541669 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:03:25.541669 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:03:25.541669 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="769c6f6763fb9f13b50d3b20e4e243095466b88fdda0ed53065e6d3a07bb1654" cmd=["/bin/sleep","15"]
Jul 09 19:03:25.542049 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.541683    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw" message="" err=<
Jul 09 19:03:25.542049 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:03:25.542049 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:03:25.542049 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:03:25.542049 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:03:25.541695    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:03:25.542049 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:03:25.542049 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:03:25.542049 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw" podUID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" containerName="main" containerID="cri-o://769c6f6763fb9f13b50d3b20e4e243095466b88fdda0ed53065e6d3a07bb1654"
Jul 09 19:03:25.542049 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.541744    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw" podUID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" containerName="main" containerID="cri-o://769c6f6763fb9f13b50d3b20e4e243095466b88fdda0ed53065e6d3a07bb1654" gracePeriod=60
Jul 09 19:03:25.771821 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.771793    2565 generic.go:356] "Generic (PLEG): container finished" podID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" containerID="769c6f6763fb9f13b50d3b20e4e243095466b88fdda0ed53065e6d3a07bb1654" exitCode=0
Jul 09 19:03:25.771964 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.771847    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw" event={"ID":"510ffc95-67a0-43dd-a57c-55bf22ccaa20","Type":"ContainerDied","Data":"769c6f6763fb9f13b50d3b20e4e243095466b88fdda0ed53065e6d3a07bb1654"}
Jul 09 19:03:25.789530 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.789471    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 19:03:25.853292 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.853260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tls-certs\") pod \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") "
Jul 09 19:03:25.853487 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.853324    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kserve-provision-location\") pod \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") "
Jul 09 19:03:25.853487 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.853342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tokenizer-uds\") pod \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") "
Jul 09 19:03:25.853609 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.853479    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj87q\" (UniqueName: \"kubernetes.io/projected/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kube-api-access-vj87q\") pod \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\" (UID: \"510ffc95-67a0-43dd-a57c-55bf22ccaa20\") "
Jul 09 19:03:25.853609 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.853596    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "510ffc95-67a0-43dd-a57c-55bf22ccaa20" (UID: "510ffc95-67a0-43dd-a57c-55bf22ccaa20"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:25.853786 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.853764    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:25.854004 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.853969    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "510ffc95-67a0-43dd-a57c-55bf22ccaa20" (UID: "510ffc95-67a0-43dd-a57c-55bf22ccaa20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:25.855538 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.855500    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kube-api-access-vj87q" (OuterVolumeSpecName: "kube-api-access-vj87q") pod "510ffc95-67a0-43dd-a57c-55bf22ccaa20" (UID: "510ffc95-67a0-43dd-a57c-55bf22ccaa20"). InnerVolumeSpecName "kube-api-access-vj87q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:03:25.855634 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.855553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "510ffc95-67a0-43dd-a57c-55bf22ccaa20" (UID: "510ffc95-67a0-43dd-a57c-55bf22ccaa20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:03:25.954514 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.954476    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj87q\" (UniqueName: \"kubernetes.io/projected/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kube-api-access-vj87q\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:25.954514 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.954507    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510ffc95-67a0-43dd-a57c-55bf22ccaa20-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:25.954514 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:25.954519    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/510ffc95-67a0-43dd-a57c-55bf22ccaa20-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:26.778552 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:26.778517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw" event={"ID":"510ffc95-67a0-43dd-a57c-55bf22ccaa20","Type":"ContainerDied","Data":"3450848066aa68a03cc70bf0d523fbb77cb33fcf91e3509b474da834d82f091b"}
Jul 09 19:03:26.779069 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:26.778552    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"
Jul 09 19:03:26.779069 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:26.778577    2565 scope.go:117] "RemoveContainer" containerID="769c6f6763fb9f13b50d3b20e4e243095466b88fdda0ed53065e6d3a07bb1654"
Jul 09 19:03:26.787872 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:26.787850    2565 scope.go:117] "RemoveContainer" containerID="f67819274d05f5ff9e0b456580be524b9fb441d489c463fc31e68c2ca1c758c9"
Jul 09 19:03:26.809335 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:26.809295    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"]
Jul 09 19:03:26.812069 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:26.812044    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw"]
Jul 09 19:03:26.869435 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:26.869398    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" path="/var/lib/kubelet/pods/510ffc95-67a0-43dd-a57c-55bf22ccaa20/volumes"
Jul 09 19:03:31.771696 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:31.771653    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:03:41.771977 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:41.771920    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:03:42.158326 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:42.158220    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 19:03:51.781843 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:51.781803    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:03:51.789441 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:51.789411    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:03:57.182257 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.182209    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="b20149cf-88f3-4d0a-b7a4-3862df357e87" containerName="main" containerID="cri-o://3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399" gracePeriod=45
Jul 09 19:03:57.862985 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.862960    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 19:03:57.934521 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.934430    2565 generic.go:356] "Generic (PLEG): container finished" podID="b20149cf-88f3-4d0a-b7a4-3862df357e87" containerID="3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399" exitCode=0
Jul 09 19:03:57.934521 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.934488    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 19:03:57.934521 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.934488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b20149cf-88f3-4d0a-b7a4-3862df357e87","Type":"ContainerDied","Data":"3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399"}
Jul 09 19:03:57.934752 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.934552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"b20149cf-88f3-4d0a-b7a4-3862df357e87","Type":"ContainerDied","Data":"f24e06b5427bcf683624972bb440c054aec65d0f45f278d07b4c615850934287"}
Jul 09 19:03:57.934752 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.934583    2565 scope.go:117] "RemoveContainer" containerID="3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399"
Jul 09 19:03:57.942717 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.942690    2565 scope.go:117] "RemoveContainer" containerID="0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb"
Jul 09 19:03:57.950717 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.950697    2565 scope.go:117] "RemoveContainer" containerID="3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399"
Jul 09 19:03:57.950994 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:03:57.950972    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399\": container with ID starting with 3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399 not found: ID does not exist" containerID="3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399"
Jul 09 19:03:57.951046 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.951003    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399"} err="failed to get container status \"3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399\": rpc error: code = NotFound desc = could not find container \"3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399\": container with ID starting with 3369805d4a335ee3a151e46de9dec38570119a06a9686229a16a762de0c98399 not found: ID does not exist"
Jul 09 19:03:57.951046 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.951021    2565 scope.go:117] "RemoveContainer" containerID="0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb"
Jul 09 19:03:57.951257 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:03:57.951239    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb\": container with ID starting with 0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb not found: ID does not exist" containerID="0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb"
Jul 09 19:03:57.951307 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.951262    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb"} err="failed to get container status \"0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb\": rpc error: code = NotFound desc = could not find container \"0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb\": container with ID starting with 0de6d7461b27417445a778132858daaa0e7c8a36266e38abc718764965e58abb not found: ID does not exist"
Jul 09 19:03:57.952490 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.952476    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-tmp-dir\") pod \"b20149cf-88f3-4d0a-b7a4-3862df357e87\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") "
Jul 09 19:03:57.952534 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.952501    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-model-cache\") pod \"b20149cf-88f3-4d0a-b7a4-3862df357e87\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") "
Jul 09 19:03:57.952574 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.952549    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg846\" (UniqueName: \"kubernetes.io/projected/b20149cf-88f3-4d0a-b7a4-3862df357e87-kube-api-access-rg846\") pod \"b20149cf-88f3-4d0a-b7a4-3862df357e87\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") "
Jul 09 19:03:57.952611 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.952590    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b20149cf-88f3-4d0a-b7a4-3862df357e87-tls-certs\") pod \"b20149cf-88f3-4d0a-b7a4-3862df357e87\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") "
Jul 09 19:03:57.952656 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.952612    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-home\") pod \"b20149cf-88f3-4d0a-b7a4-3862df357e87\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") "
Jul 09 19:03:57.952808 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.952788    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-dshm\") pod \"b20149cf-88f3-4d0a-b7a4-3862df357e87\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") "
Jul 09 19:03:57.953012 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.952832    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-model-cache" (OuterVolumeSpecName: "model-cache") pod "b20149cf-88f3-4d0a-b7a4-3862df357e87" (UID: "b20149cf-88f3-4d0a-b7a4-3862df357e87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:57.953012 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.952866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-kserve-provision-location\") pod \"b20149cf-88f3-4d0a-b7a4-3862df357e87\" (UID: \"b20149cf-88f3-4d0a-b7a4-3862df357e87\") "
Jul 09 19:03:57.953246 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.953214    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:57.953642 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.953618    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-home" (OuterVolumeSpecName: "home") pod "b20149cf-88f3-4d0a-b7a4-3862df357e87" (UID: "b20149cf-88f3-4d0a-b7a4-3862df357e87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:57.955252 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.955203    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b20149cf-88f3-4d0a-b7a4-3862df357e87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b20149cf-88f3-4d0a-b7a4-3862df357e87" (UID: "b20149cf-88f3-4d0a-b7a4-3862df357e87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:03:57.955252 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.955242    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-dshm" (OuterVolumeSpecName: "dshm") pod "b20149cf-88f3-4d0a-b7a4-3862df357e87" (UID: "b20149cf-88f3-4d0a-b7a4-3862df357e87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:57.955798 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.955774    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20149cf-88f3-4d0a-b7a4-3862df357e87-kube-api-access-rg846" (OuterVolumeSpecName: "kube-api-access-rg846") pod "b20149cf-88f3-4d0a-b7a4-3862df357e87" (UID: "b20149cf-88f3-4d0a-b7a4-3862df357e87"). InnerVolumeSpecName "kube-api-access-rg846". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:03:57.964270 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:57.964245    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b20149cf-88f3-4d0a-b7a4-3862df357e87" (UID: "b20149cf-88f3-4d0a-b7a4-3862df357e87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:58.010649 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.010592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b20149cf-88f3-4d0a-b7a4-3862df357e87" (UID: "b20149cf-88f3-4d0a-b7a4-3862df357e87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:58.054472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.054427    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg846\" (UniqueName: \"kubernetes.io/projected/b20149cf-88f3-4d0a-b7a4-3862df357e87-kube-api-access-rg846\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:58.054472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.054460    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b20149cf-88f3-4d0a-b7a4-3862df357e87-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:58.054472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.054471    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:58.054472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.054481    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:58.054761 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.054488    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:58.054761 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.054501    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b20149cf-88f3-4d0a-b7a4-3862df357e87-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:03:58.257899 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.257863    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 19:03:58.262320 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.262295    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 19:03:58.867742 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:03:58.867712    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b20149cf-88f3-4d0a-b7a4-3862df357e87" path="/var/lib/kubelet/pods/b20149cf-88f3-4d0a-b7a4-3862df357e87/volumes"
Jul 09 19:04:40.976080 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:40.975971    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:04:40.983449 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:40.983425    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:04:46.057146 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057107    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"]
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057430    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" containerName="storage-initializer"
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057443    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" containerName="storage-initializer"
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057463    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" containerName="main"
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057470    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" containerName="main"
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057476    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b20149cf-88f3-4d0a-b7a4-3862df357e87" containerName="main"
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057482    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b20149cf-88f3-4d0a-b7a4-3862df357e87" containerName="main"
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057493    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b20149cf-88f3-4d0a-b7a4-3862df357e87" containerName="storage-initializer"
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057498    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b20149cf-88f3-4d0a-b7a4-3862df357e87" containerName="storage-initializer"
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057548    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b20149cf-88f3-4d0a-b7a4-3862df357e87" containerName="main"
Jul 09 19:04:46.057563 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.057557    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="510ffc95-67a0-43dd-a57c-55bf22ccaa20" containerName="main"
Jul 09 19:04:46.060534 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.060517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.063047 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.063024    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 09 19:04:46.070777 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.070749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"]
Jul 09 19:04:46.189094 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.189054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krbg\" (UniqueName: \"kubernetes.io/projected/955d74a7-ea6e-4eb4-8357-5cace7695106-kube-api-access-8krbg\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.189287 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.189111    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.189287 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.189178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.189287 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.189212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.189287 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.189234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d74a7-ea6e-4eb4-8357-5cace7695106-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.189287 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.189266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.290626 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.290594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.290626 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.290633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.290889 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.290649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d74a7-ea6e-4eb4-8357-5cace7695106-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.290889 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.290676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.290889 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.290719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krbg\" (UniqueName: \"kubernetes.io/projected/955d74a7-ea6e-4eb4-8357-5cace7695106-kube-api-access-8krbg\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.290889 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.290751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.291140 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.291099    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.291186 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.291137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.291222 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.291203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.292925 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.292892    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.293360 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.293340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d74a7-ea6e-4eb4-8357-5cace7695106-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.298717 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.298699    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krbg\" (UniqueName: \"kubernetes.io/projected/955d74a7-ea6e-4eb4-8357-5cace7695106-kube-api-access-8krbg\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.371933 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.371827    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:46.484686 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.484649    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"]
Jul 09 19:04:46.489341 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.489314    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.492089 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.492065    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-q29xp\""
Jul 09 19:04:46.497814 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.497742    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"]
Jul 09 19:04:46.513914 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.513879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"]
Jul 09 19:04:46.516536 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:04:46.516511    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955d74a7_ea6e_4eb4_8357_5cace7695106.slice/crio-609b71da77d57f098eff487deccbc6dad7098b357b812e67c61a58919241c178 WatchSource:0}: Error finding container 609b71da77d57f098eff487deccbc6dad7098b357b812e67c61a58919241c178: Status 404 returned error can't find the container with id 609b71da77d57f098eff487deccbc6dad7098b357b812e67c61a58919241c178
Jul 09 19:04:46.592818 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.592790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a962e9-20f3-4bd8-a695-a2f056e58f32-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.592981 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.592834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqdlb\" (UniqueName: \"kubernetes.io/projected/63a962e9-20f3-4bd8-a695-a2f056e58f32-kube-api-access-vqdlb\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.592981 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.592898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/63a962e9-20f3-4bd8-a695-a2f056e58f32-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.693797 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.693763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/63a962e9-20f3-4bd8-a695-a2f056e58f32-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.693883 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.693868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a962e9-20f3-4bd8-a695-a2f056e58f32-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.693951 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.693932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqdlb\" (UniqueName: \"kubernetes.io/projected/63a962e9-20f3-4bd8-a695-a2f056e58f32-kube-api-access-vqdlb\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.694170 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.694151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/63a962e9-20f3-4bd8-a695-a2f056e58f32-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.696273 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.696248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a962e9-20f3-4bd8-a695-a2f056e58f32-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.702980 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.702958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqdlb\" (UniqueName: \"kubernetes.io/projected/63a962e9-20f3-4bd8-a695-a2f056e58f32-kube-api-access-vqdlb\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.801530 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.801497    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:46.945888 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:46.945862    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"]
Jul 09 19:04:46.948404 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:04:46.948378    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a962e9_20f3_4bd8_a695_a2f056e58f32.slice/crio-205328493ba1d709b4544c87929ed0feb2e9e07ca4fbcc7d4554dc06a7db848b WatchSource:0}: Error finding container 205328493ba1d709b4544c87929ed0feb2e9e07ca4fbcc7d4554dc06a7db848b: Status 404 returned error can't find the container with id 205328493ba1d709b4544c87929ed0feb2e9e07ca4fbcc7d4554dc06a7db848b
Jul 09 19:04:47.163490 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:47.163450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj" event={"ID":"955d74a7-ea6e-4eb4-8357-5cace7695106","Type":"ContainerStarted","Data":"b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5"}
Jul 09 19:04:47.163982 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:47.163518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj" event={"ID":"955d74a7-ea6e-4eb4-8357-5cace7695106","Type":"ContainerStarted","Data":"609b71da77d57f098eff487deccbc6dad7098b357b812e67c61a58919241c178"}
Jul 09 19:04:47.164866 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:47.164845    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9" event={"ID":"63a962e9-20f3-4bd8-a695-a2f056e58f32","Type":"ContainerStarted","Data":"a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd"}
Jul 09 19:04:47.164944 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:47.164883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9" event={"ID":"63a962e9-20f3-4bd8-a695-a2f056e58f32","Type":"ContainerStarted","Data":"205328493ba1d709b4544c87929ed0feb2e9e07ca4fbcc7d4554dc06a7db848b"}
Jul 09 19:04:47.164944 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:47.164916    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:04:47.200421 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:47.200324    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj" podStartSLOduration=1.20030748 podStartE2EDuration="1.20030748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:04:47.199280641 +0000 UTC m=+3606.917543406" watchObservedRunningTime="2026-07-09 19:04:47.20030748 +0000 UTC m=+3606.918570243"
Jul 09 19:04:47.271458 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:47.271403    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9" podStartSLOduration=1.271383536 podStartE2EDuration="1.271383536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:04:47.267013841 +0000 UTC m=+3606.985276607" watchObservedRunningTime="2026-07-09 19:04:47.271383536 +0000 UTC m=+3606.989646300"
Jul 09 19:04:47.372738 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:47.372706    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:51.139631 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:51.139596    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 19:04:56.372631 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:56.372590    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:56.385248 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:56.385215    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:04:57.219768 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:04:57.219727    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:05:06.163774 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.163735    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" containerID="cri-o://c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1" gracePeriod=45
Jul 09 19:05:06.782592 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.782558    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 09 19:05:06.938368 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.938305    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:05:06.973623 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.973590    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-tmp-dir\") pod \"7942d565-ff47-4f37-9860-c62d39548388\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") "
Jul 09 19:05:06.973806 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.973674    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-model-cache\") pod \"7942d565-ff47-4f37-9860-c62d39548388\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") "
Jul 09 19:05:06.973806 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.973701    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-dshm\") pod \"7942d565-ff47-4f37-9860-c62d39548388\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") "
Jul 09 19:05:06.973806 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.973743    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjjcb\" (UniqueName: \"kubernetes.io/projected/7942d565-ff47-4f37-9860-c62d39548388-kube-api-access-gjjcb\") pod \"7942d565-ff47-4f37-9860-c62d39548388\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") "
Jul 09 19:05:06.973806 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.973785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-home\") pod \"7942d565-ff47-4f37-9860-c62d39548388\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") "
Jul 09 19:05:06.974050 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.973812    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7942d565-ff47-4f37-9860-c62d39548388-tls-certs\") pod \"7942d565-ff47-4f37-9860-c62d39548388\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") "
Jul 09 19:05:06.974050 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.973924    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") pod \"7942d565-ff47-4f37-9860-c62d39548388\" (UID: \"7942d565-ff47-4f37-9860-c62d39548388\") "
Jul 09 19:05:06.974050 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.974039    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-model-cache" (OuterVolumeSpecName: "model-cache") pod "7942d565-ff47-4f37-9860-c62d39548388" (UID: "7942d565-ff47-4f37-9860-c62d39548388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:05:06.974308 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.974285    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:05:06.974503 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.974475    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-home" (OuterVolumeSpecName: "home") pod "7942d565-ff47-4f37-9860-c62d39548388" (UID: "7942d565-ff47-4f37-9860-c62d39548388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:05:06.976505 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.976478    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7942d565-ff47-4f37-9860-c62d39548388-kube-api-access-gjjcb" (OuterVolumeSpecName: "kube-api-access-gjjcb") pod "7942d565-ff47-4f37-9860-c62d39548388" (UID: "7942d565-ff47-4f37-9860-c62d39548388"). InnerVolumeSpecName "kube-api-access-gjjcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:05:06.976714 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.976686    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-dshm" (OuterVolumeSpecName: "dshm") pod "7942d565-ff47-4f37-9860-c62d39548388" (UID: "7942d565-ff47-4f37-9860-c62d39548388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:05:06.976862 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.976805    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7942d565-ff47-4f37-9860-c62d39548388-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7942d565-ff47-4f37-9860-c62d39548388" (UID: "7942d565-ff47-4f37-9860-c62d39548388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:05:06.977418 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.977397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564" (OuterVolumeSpecName: "kserve-pvc-source") pod "7942d565-ff47-4f37-9860-c62d39548388" (UID: "7942d565-ff47-4f37-9860-c62d39548388"). InnerVolumeSpecName "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 19:05:06.994326 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:06.994289    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7942d565-ff47-4f37-9860-c62d39548388" (UID: "7942d565-ff47-4f37-9860-c62d39548388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:05:07.075580 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.075540    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.075580 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.075572    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjcb\" (UniqueName: \"kubernetes.io/projected/7942d565-ff47-4f37-9860-c62d39548388-kube-api-access-gjjcb\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.075580 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.075583    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.075580 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.075591    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7942d565-ff47-4f37-9860-c62d39548388-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.075962 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.075620    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") on node \"ip-10-0-129-87.ec2.internal\" "
Jul 09 19:05:07.075962 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.075628    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7942d565-ff47-4f37-9860-c62d39548388-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:05:07.103199 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.103177    2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564") on node "ip-10-0-129-87.ec2.internal"
Jul 09 19:05:07.176090 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.176054    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-2525289e-aa0d-4f3c-83d9-c379852c234b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f3be72c734b32564\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"csi-55bd14af576e77d6436311c1b9c2889f6970de0650f144b747e86982125a0cd9\""
Jul 09 19:05:07.257779 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.257740    2565 generic.go:356] "Generic (PLEG): container finished" podID="7942d565-ff47-4f37-9860-c62d39548388" containerID="c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1" exitCode=0
Jul 09 19:05:07.257957 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.257795    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 19:05:07.257957 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.257825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"7942d565-ff47-4f37-9860-c62d39548388","Type":"ContainerDied","Data":"c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1"}
Jul 09 19:05:07.257957 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.257885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"7942d565-ff47-4f37-9860-c62d39548388","Type":"ContainerDied","Data":"f3c0cb1ed42f2c182ee8ecaf2f04ec4c345eb59b66a7f89a824d25a3d9dde93f"}
Jul 09 19:05:07.257957 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.257925    2565 scope.go:117] "RemoveContainer" containerID="c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1"
Jul 09 19:05:07.267030 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.267015    2565 scope.go:117] "RemoveContainer" containerID="c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1"
Jul 09 19:05:07.267277 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:05:07.267256    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1\": container with ID starting with c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1 not found: ID does not exist" containerID="c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1"
Jul 09 19:05:07.267326 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.267288    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1"} err="failed to get container status \"c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1\": rpc error: code = NotFound desc = could not find container \"c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1\": container with ID starting with c11e59bd4aaf8e6a308a2770f1dd3951ca71b17e50577b8c3779e8c4ee66b4c1 not found: ID does not exist"
Jul 09 19:05:07.280994 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.280967    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 19:05:07.288986 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:07.288954    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 19:05:08.869324 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:08.869289    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7942d565-ff47-4f37-9860-c62d39548388" path="/var/lib/kubelet/pods/7942d565-ff47-4f37-9860-c62d39548388/volumes"
Jul 09 19:05:18.171592 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:05:18.171560    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:06:03.498581 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.498487    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"]
Jul 09 19:06:03.499067 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.499032    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main"
Jul 09 19:06:03.499067 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.499052    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main"
Jul 09 19:06:03.499194 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.499156    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7942d565-ff47-4f37-9860-c62d39548388" containerName="main"
Jul 09 19:06:03.502238 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.502215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.505233 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.505208    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 19:06:03.505340 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.505217    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 19:06:03.516466 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.516441    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"]
Jul 09 19:06:03.578474 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.578435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kserve-provision-location\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.578668 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.578478    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tmp-dir\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.578668 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.578537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-model-cache\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.578668 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.578615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcfwv\" (UniqueName: \"kubernetes.io/projected/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kube-api-access-mcfwv\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.578668 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.578658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tls-certs\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.578890 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.578694    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-dshm\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.578890 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.578785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-home\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.679544 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.679509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-home\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.679745 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.679554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kserve-provision-location\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.679745 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.679576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tmp-dir\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.679745 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.679610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-model-cache\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.679745 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.679637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcfwv\" (UniqueName: \"kubernetes.io/projected/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kube-api-access-mcfwv\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.679745 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.679680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tls-certs\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.679745 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.679706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-dshm\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.680124 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.680008    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tmp-dir\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.680124 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.680042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-home\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.680227 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.680140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kserve-provision-location\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.680278 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.680256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-model-cache\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.682133 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.682112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-dshm\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.682532 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.682507    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tls-certs\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.689566 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.689540    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcfwv\" (UniqueName: \"kubernetes.io/projected/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kube-api-access-mcfwv\") pod \"stop-feature-test-kserve-5c675d88cb-pgk2n\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.813101 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.813013    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:03.916262 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.916231    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"]
Jul 09 19:06:03.920198 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.920175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:03.923122 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.923102    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-bdv75\""
Jul 09 19:06:03.939932 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.939887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"]
Jul 09 19:06:03.968467 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.968431    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"]
Jul 09 19:06:03.971389 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:06:03.971356    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96d804d_feb5_4c1f_a900_d16ff1afcd83.slice/crio-e58cee8a8c4cd24da9e2444a5144c43084fc30f10051fb51992b7f7105f7e7a3 WatchSource:0}: Error finding container e58cee8a8c4cd24da9e2444a5144c43084fc30f10051fb51992b7f7105f7e7a3: Status 404 returned error can't find the container with id e58cee8a8c4cd24da9e2444a5144c43084fc30f10051fb51992b7f7105f7e7a3
Jul 09 19:06:03.981464 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.981442    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:03.981550 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.981479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mh9\" (UniqueName: \"kubernetes.io/projected/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kube-api-access-j8mh9\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:03.981598 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.981537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:03.981598 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:03.981582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.082787 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.082697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.082787 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.082743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mh9\" (UniqueName: \"kubernetes.io/projected/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kube-api-access-j8mh9\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.083027 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.082894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.083027 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.082963    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.083271 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.083254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.083332 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.083276    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.085237 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.085217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.092256 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.092231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mh9\" (UniqueName: \"kubernetes.io/projected/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kube-api-access-j8mh9\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.232044 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.232013    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:04.387185 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.387158    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"]
Jul 09 19:06:04.388682 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:06:04.388654    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09281d2_36ca_43fa_9d3d_0153ae7d5b09.slice/crio-faf72cded59fe1f18f5d4f4d202868220f499451660b1657ddd9ee8c204b0d22 WatchSource:0}: Error finding container faf72cded59fe1f18f5d4f4d202868220f499451660b1657ddd9ee8c204b0d22: Status 404 returned error can't find the container with id faf72cded59fe1f18f5d4f4d202868220f499451660b1657ddd9ee8c204b0d22
Jul 09 19:06:04.529227 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.529182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" event={"ID":"a09281d2-36ca-43fa-9d3d-0153ae7d5b09","Type":"ContainerStarted","Data":"dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac"}
Jul 09 19:06:04.529711 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.529242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" event={"ID":"a09281d2-36ca-43fa-9d3d-0153ae7d5b09","Type":"ContainerStarted","Data":"faf72cded59fe1f18f5d4f4d202868220f499451660b1657ddd9ee8c204b0d22"}
Jul 09 19:06:04.530835 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.530810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" event={"ID":"b96d804d-feb5-4c1f-a900-d16ff1afcd83","Type":"ContainerStarted","Data":"5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29"}
Jul 09 19:06:04.530988 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:04.530857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" event={"ID":"b96d804d-feb5-4c1f-a900-d16ff1afcd83","Type":"ContainerStarted","Data":"e58cee8a8c4cd24da9e2444a5144c43084fc30f10051fb51992b7f7105f7e7a3"}
Jul 09 19:06:05.538767 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:05.538726    2565 generic.go:356] "Generic (PLEG): container finished" podID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" containerID="dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac" exitCode=0
Jul 09 19:06:05.539273 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:05.538816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" event={"ID":"a09281d2-36ca-43fa-9d3d-0153ae7d5b09","Type":"ContainerDied","Data":"dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac"}
Jul 09 19:06:06.546969 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:06.546929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" event={"ID":"a09281d2-36ca-43fa-9d3d-0153ae7d5b09","Type":"ContainerStarted","Data":"44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d"}
Jul 09 19:06:06.547493 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:06.547042    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:06.569088 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:06.569005    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" podStartSLOduration=2.603841202 podStartE2EDuration="3.568987388s" totalImagesPullingTime="0s" totalInitContainerRuntime="965.146186ms" isStatefulPod=false podCreationTimestamp="2026-07-09 19:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:06:06.568471662 +0000 UTC m=+3686.286734437" watchObservedRunningTime="2026-07-09 19:06:06.568987388 +0000 UTC m=+3686.287250156"
Jul 09 19:06:08.558129 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:08.558094    2565 generic.go:356] "Generic (PLEG): container finished" podID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerID="5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29" exitCode=0
Jul 09 19:06:08.558517 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:08.558170    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" event={"ID":"b96d804d-feb5-4c1f-a900-d16ff1afcd83","Type":"ContainerDied","Data":"5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29"}
Jul 09 19:06:09.565293 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:09.565251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" event={"ID":"b96d804d-feb5-4c1f-a900-d16ff1afcd83","Type":"ContainerStarted","Data":"db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2"}
Jul 09 19:06:09.590173 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:09.590098    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podStartSLOduration=2.5891153080000002 podStartE2EDuration="6.590073458s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.00095815s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:06:09.588386584 +0000 UTC m=+3689.306649349" watchObservedRunningTime="2026-07-09 19:06:09.590073458 +0000 UTC m=+3689.308336222"
Jul 09 19:06:09.813413 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:09.813370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:13.813649 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:13.813606    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:06:13.815213 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:13.815177    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:06:23.813891 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:23.813852    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:06:28.651685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:28.651617    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"]
Jul 09 19:06:28.655194 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:28.655163    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"]
Jul 09 19:06:28.662097 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:06:28.662047    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:06:28.662097 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:06:28.662097 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:06:28.662097 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd" cmd=["/bin/sleep","15"]
Jul 09 19:06:28.662429 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:28.662186    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9" message="" err=<
Jul 09 19:06:28.662429 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:06:28.662429 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:06:28.662429 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:06:28.662429 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:06:28.662226    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:06:28.662429 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:06:28.662429 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:06:28.662429 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9" podUID="63a962e9-20f3-4bd8-a695-a2f056e58f32" containerName="main" containerID="cri-o://a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd"
Jul 09 19:06:28.662429 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:28.662272    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9" podUID="63a962e9-20f3-4bd8-a695-a2f056e58f32" containerName="main" containerID="cri-o://a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd" gracePeriod=60
Jul 09 19:06:28.931711 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:28.931686    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:06:29.027587 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.027546    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqdlb\" (UniqueName: \"kubernetes.io/projected/63a962e9-20f3-4bd8-a695-a2f056e58f32-kube-api-access-vqdlb\") pod \"63a962e9-20f3-4bd8-a695-a2f056e58f32\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") "
Jul 09 19:06:29.027774 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.027629    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a962e9-20f3-4bd8-a695-a2f056e58f32-tls-certs\") pod \"63a962e9-20f3-4bd8-a695-a2f056e58f32\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") "
Jul 09 19:06:29.027774 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.027679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/63a962e9-20f3-4bd8-a695-a2f056e58f32-tokenizer-uds\") pod \"63a962e9-20f3-4bd8-a695-a2f056e58f32\" (UID: \"63a962e9-20f3-4bd8-a695-a2f056e58f32\") "
Jul 09 19:06:29.028083 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.028048    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a962e9-20f3-4bd8-a695-a2f056e58f32-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "63a962e9-20f3-4bd8-a695-a2f056e58f32" (UID: "63a962e9-20f3-4bd8-a695-a2f056e58f32"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:06:29.029704 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.029682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63a962e9-20f3-4bd8-a695-a2f056e58f32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63a962e9-20f3-4bd8-a695-a2f056e58f32" (UID: "63a962e9-20f3-4bd8-a695-a2f056e58f32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:06:29.029810 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.029723    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a962e9-20f3-4bd8-a695-a2f056e58f32-kube-api-access-vqdlb" (OuterVolumeSpecName: "kube-api-access-vqdlb") pod "63a962e9-20f3-4bd8-a695-a2f056e58f32" (UID: "63a962e9-20f3-4bd8-a695-a2f056e58f32"). InnerVolumeSpecName "kube-api-access-vqdlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:06:29.129359 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.129314    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqdlb\" (UniqueName: \"kubernetes.io/projected/63a962e9-20f3-4bd8-a695-a2f056e58f32-kube-api-access-vqdlb\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:06:29.129359 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.129359    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a962e9-20f3-4bd8-a695-a2f056e58f32-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:06:29.129612 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.129374    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/63a962e9-20f3-4bd8-a695-a2f056e58f32-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:06:29.670263 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.670224    2565 generic.go:356] "Generic (PLEG): container finished" podID="63a962e9-20f3-4bd8-a695-a2f056e58f32" containerID="a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd" exitCode=0
Jul 09 19:06:29.670700 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.670280    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"
Jul 09 19:06:29.670700 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.670311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9" event={"ID":"63a962e9-20f3-4bd8-a695-a2f056e58f32","Type":"ContainerDied","Data":"a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd"}
Jul 09 19:06:29.670700 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.670361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9" event={"ID":"63a962e9-20f3-4bd8-a695-a2f056e58f32","Type":"ContainerDied","Data":"205328493ba1d709b4544c87929ed0feb2e9e07ca4fbcc7d4554dc06a7db848b"}
Jul 09 19:06:29.670700 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.670383    2565 scope.go:117] "RemoveContainer" containerID="a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd"
Jul 09 19:06:29.679248 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.679227    2565 scope.go:117] "RemoveContainer" containerID="a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd"
Jul 09 19:06:29.679557 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:06:29.679532    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd\": container with ID starting with a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd not found: ID does not exist" containerID="a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd"
Jul 09 19:06:29.679617 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.679569    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd"} err="failed to get container status \"a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd\": rpc error: code = NotFound desc = could not find container \"a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd\": container with ID starting with a3378a10a563b56cbfe743fb9285ea1e934e3355cb5237a91449790dee33d7bd not found: ID does not exist"
Jul 09 19:06:29.696357 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.696322    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"]
Jul 09 19:06:29.702106 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:29.702083    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddft5zs9"]
Jul 09 19:06:30.868477 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:30.868448    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63a962e9-20f3-4bd8-a695-a2f056e58f32" path="/var/lib/kubelet/pods/63a962e9-20f3-4bd8-a695-a2f056e58f32/volumes"
Jul 09 19:06:33.814218 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:33.814170    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:06:37.553964 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:37.553933    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:06:43.682327 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:43.682260    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj" podUID="955d74a7-ea6e-4eb4-8357-5cace7695106" containerName="main" containerID="cri-o://b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5" gracePeriod=45
Jul 09 19:06:43.813822 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:43.813766    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:06:43.942568 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:43.942500    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:06:44.071917 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.071864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-model-cache\") pod \"955d74a7-ea6e-4eb4-8357-5cace7695106\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") "
Jul 09 19:06:44.072096 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.071947    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-dshm\") pod \"955d74a7-ea6e-4eb4-8357-5cace7695106\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") "
Jul 09 19:06:44.072096 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.071985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-tmp-dir\") pod \"955d74a7-ea6e-4eb4-8357-5cace7695106\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") "
Jul 09 19:06:44.072096 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.072013    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d74a7-ea6e-4eb4-8357-5cace7695106-tls-certs\") pod \"955d74a7-ea6e-4eb4-8357-5cace7695106\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") "
Jul 09 19:06:44.072096 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.072057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-home\") pod \"955d74a7-ea6e-4eb4-8357-5cace7695106\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") "
Jul 09 19:06:44.072313 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.072096    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8krbg\" (UniqueName: \"kubernetes.io/projected/955d74a7-ea6e-4eb4-8357-5cace7695106-kube-api-access-8krbg\") pod \"955d74a7-ea6e-4eb4-8357-5cace7695106\" (UID: \"955d74a7-ea6e-4eb4-8357-5cace7695106\") "
Jul 09 19:06:44.072313 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.072226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-model-cache" (OuterVolumeSpecName: "model-cache") pod "955d74a7-ea6e-4eb4-8357-5cace7695106" (UID: "955d74a7-ea6e-4eb4-8357-5cace7695106"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:06:44.072407 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.072315    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "955d74a7-ea6e-4eb4-8357-5cace7695106" (UID: "955d74a7-ea6e-4eb4-8357-5cace7695106"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:06:44.072472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.072453    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:06:44.072525 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.072473    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:06:44.072572 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.072537    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-home" (OuterVolumeSpecName: "home") pod "955d74a7-ea6e-4eb4-8357-5cace7695106" (UID: "955d74a7-ea6e-4eb4-8357-5cace7695106"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:06:44.074204 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.074170    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-dshm" (OuterVolumeSpecName: "dshm") pod "955d74a7-ea6e-4eb4-8357-5cace7695106" (UID: "955d74a7-ea6e-4eb4-8357-5cace7695106"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:06:44.074360 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.074337    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955d74a7-ea6e-4eb4-8357-5cace7695106-kube-api-access-8krbg" (OuterVolumeSpecName: "kube-api-access-8krbg") pod "955d74a7-ea6e-4eb4-8357-5cace7695106" (UID: "955d74a7-ea6e-4eb4-8357-5cace7695106"). InnerVolumeSpecName "kube-api-access-8krbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:06:44.074404 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.074337    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955d74a7-ea6e-4eb4-8357-5cace7695106-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "955d74a7-ea6e-4eb4-8357-5cace7695106" (UID: "955d74a7-ea6e-4eb4-8357-5cace7695106"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:06:44.173418 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.173375    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8krbg\" (UniqueName: \"kubernetes.io/projected/955d74a7-ea6e-4eb4-8357-5cace7695106-kube-api-access-8krbg\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:06:44.173418 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.173409    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:06:44.173418 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.173419    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d74a7-ea6e-4eb4-8357-5cace7695106-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:06:44.173418 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.173428    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d74a7-ea6e-4eb4-8357-5cace7695106-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:06:44.756207 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.756169    2565 generic.go:356] "Generic (PLEG): container finished" podID="955d74a7-ea6e-4eb4-8357-5cace7695106" containerID="b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5" exitCode=0
Jul 09 19:06:44.756207 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.756208    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"
Jul 09 19:06:44.756689 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.756246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj" event={"ID":"955d74a7-ea6e-4eb4-8357-5cace7695106","Type":"ContainerDied","Data":"b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5"}
Jul 09 19:06:44.756689 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.756305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj" event={"ID":"955d74a7-ea6e-4eb4-8357-5cace7695106","Type":"ContainerDied","Data":"609b71da77d57f098eff487deccbc6dad7098b357b812e67c61a58919241c178"}
Jul 09 19:06:44.756689 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.756327    2565 scope.go:117] "RemoveContainer" containerID="b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5"
Jul 09 19:06:44.766272 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.766251    2565 scope.go:117] "RemoveContainer" containerID="b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5"
Jul 09 19:06:44.766579 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:06:44.766561    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5\": container with ID starting with b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5 not found: ID does not exist" containerID="b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5"
Jul 09 19:06:44.766631 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.766588    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5"} err="failed to get container status \"b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5\": rpc error: code = NotFound desc = could not find container \"b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5\": container with ID starting with b0c18edb2759fb925aa9f34296824f2df5f69ad3160b3de500647a6434ac90a5 not found: ID does not exist"
Jul 09 19:06:44.789357 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.789321    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"]
Jul 09 19:06:44.791025 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.791003    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-jjkmj"]
Jul 09 19:06:44.868797 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:44.868761    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="955d74a7-ea6e-4eb4-8357-5cace7695106" path="/var/lib/kubelet/pods/955d74a7-ea6e-4eb4-8357-5cace7695106/volumes"
Jul 09 19:06:53.814246 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:06:53.814202    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:07:03.814149 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:03.814094    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:07:13.814377 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:13.814325    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:07:23.813608 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:23.813566    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:07:25.281015 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.280984    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"]
Jul 09 19:07:25.281416 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.281316    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63a962e9-20f3-4bd8-a695-a2f056e58f32" containerName="main"
Jul 09 19:07:25.281416 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.281327    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="63a962e9-20f3-4bd8-a695-a2f056e58f32" containerName="main"
Jul 09 19:07:25.281416 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.281339    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955d74a7-ea6e-4eb4-8357-5cace7695106" containerName="main"
Jul 09 19:07:25.281416 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.281345    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="955d74a7-ea6e-4eb4-8357-5cace7695106" containerName="main"
Jul 09 19:07:25.281416 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.281402    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="955d74a7-ea6e-4eb4-8357-5cace7695106" containerName="main"
Jul 09 19:07:25.281416 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.281413    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="63a962e9-20f3-4bd8-a695-a2f056e58f32" containerName="main"
Jul 09 19:07:25.283328 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.283309    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.287223 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.287202    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 09 19:07:25.287350 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.287220    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-h7s4m\""
Jul 09 19:07:25.295942 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.295888    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"]
Jul 09 19:07:25.348650 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.348602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.348650 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.348647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwlp\" (UniqueName: \"kubernetes.io/projected/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-kube-api-access-qgwlp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.348959 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.348685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.450028 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.449982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.450248 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.450104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.450248 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.450144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwlp\" (UniqueName: \"kubernetes.io/projected/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-kube-api-access-qgwlp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.450459 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.450434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.452804 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.452764    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.459594 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.459570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwlp\" (UniqueName: \"kubernetes.io/projected/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-kube-api-access-qgwlp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.593560 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.593461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.729210 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.729182    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"]
Jul 09 19:07:25.732153 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:07:25.732123    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067fee28_f9e5_4ad6_acd2_9b2cb8f7a401.slice/crio-2578f220bdc7242f01cc8be267c468d2a509e2cbbc2d8ab54837998fecdaaeb8 WatchSource:0}: Error finding container 2578f220bdc7242f01cc8be267c468d2a509e2cbbc2d8ab54837998fecdaaeb8: Status 404 returned error can't find the container with id 2578f220bdc7242f01cc8be267c468d2a509e2cbbc2d8ab54837998fecdaaeb8
Jul 09 19:07:25.953435 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.953401    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd" event={"ID":"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401","Type":"ContainerStarted","Data":"dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0"}
Jul 09 19:07:25.953599 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.953456    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd" event={"ID":"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401","Type":"ContainerStarted","Data":"2578f220bdc7242f01cc8be267c468d2a509e2cbbc2d8ab54837998fecdaaeb8"}
Jul 09 19:07:25.953599 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.953487    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:07:25.972198 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:25.972149    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd" podStartSLOduration=0.972132208 podStartE2EDuration="972.132208ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:25.971037834 +0000 UTC m=+3765.689300621" watchObservedRunningTime="2026-07-09 19:07:25.972132208 +0000 UTC m=+3765.690394972"
Jul 09 19:07:33.814278 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:33.814179    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:07:43.813893 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:43.813848    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:07:53.814425 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:53.814365    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Jul 09 19:07:56.960385 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:07:56.960351    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:08:03.823568 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:03.823536    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:08:03.831478 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:03.831456    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:08:51.516928 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.516875    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"]
Jul 09 19:08:51.526624 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:08:51.526587    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:08:51.526624 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:08:51.526624 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:08:51.526624 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0" cmd=["/bin/sleep","15"]
Jul 09 19:08:51.526993 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.526644    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd" message="" err=<
Jul 09 19:08:51.526993 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:08:51.526993 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:08:51.526993 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:08:51.526993 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:08:51.526661    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:08:51.526993 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:08:51.526993 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:08:51.526993 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd" podUID="067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" containerName="main" containerID="cri-o://dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0"
Jul 09 19:08:51.526993 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.526718    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd" podUID="067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" containerName="main" containerID="cri-o://dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0" gracePeriod=60
Jul 09 19:08:51.782384 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.782355    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:08:51.831806 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.831773    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tls-certs\") pod \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") "
Jul 09 19:08:51.832023 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.831826    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tokenizer-uds\") pod \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") "
Jul 09 19:08:51.832023 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.831949    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgwlp\" (UniqueName: \"kubernetes.io/projected/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-kube-api-access-qgwlp\") pod \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\" (UID: \"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401\") "
Jul 09 19:08:51.832166 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.832130    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" (UID: "067fee28-f9e5-4ad6-acd2-9b2cb8f7a401"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:08:51.832288 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.832266    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:08:51.834147 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.834114    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-kube-api-access-qgwlp" (OuterVolumeSpecName: "kube-api-access-qgwlp") pod "067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" (UID: "067fee28-f9e5-4ad6-acd2-9b2cb8f7a401"). InnerVolumeSpecName "kube-api-access-qgwlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:08:51.834147 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.834130    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" (UID: "067fee28-f9e5-4ad6-acd2-9b2cb8f7a401"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:08:51.932769 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.932735    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:08:51.932769 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.932764    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgwlp\" (UniqueName: \"kubernetes.io/projected/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401-kube-api-access-qgwlp\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:08:51.933251 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:51.933227    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"]
Jul 09 19:08:52.370104 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.370061    2565 generic.go:356] "Generic (PLEG): container finished" podID="067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" containerID="dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0" exitCode=0
Jul 09 19:08:52.370283 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.370124    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"
Jul 09 19:08:52.370283 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.370147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd" event={"ID":"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401","Type":"ContainerDied","Data":"dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0"}
Jul 09 19:08:52.370283 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.370229    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd" event={"ID":"067fee28-f9e5-4ad6-acd2-9b2cb8f7a401","Type":"ContainerDied","Data":"2578f220bdc7242f01cc8be267c468d2a509e2cbbc2d8ab54837998fecdaaeb8"}
Jul 09 19:08:52.370283 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.370250    2565 scope.go:117] "RemoveContainer" containerID="dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0"
Jul 09 19:08:52.379697 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.379678    2565 scope.go:117] "RemoveContainer" containerID="dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0"
Jul 09 19:08:52.380010 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:08:52.379991    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0\": container with ID starting with dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0 not found: ID does not exist" containerID="dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0"
Jul 09 19:08:52.380072 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.380018    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0"} err="failed to get container status \"dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0\": rpc error: code = NotFound desc = could not find container \"dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0\": container with ID starting with dbe3a1534d53fe8c1ca8f69e2ad24b345da3894fea38ebab5493868b30a7e5d0 not found: ID does not exist"
Jul 09 19:08:52.394313 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.394277    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"]
Jul 09 19:08:52.399675 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.399647    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schefxncd"]
Jul 09 19:08:52.868361 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:52.868318    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" path="/var/lib/kubelet/pods/067fee28-f9e5-4ad6-acd2-9b2cb8f7a401/volumes"
Jul 09 19:08:53.337483 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.337449    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"]
Jul 09 19:08:53.346877 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:08:53.346840    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:08:53.346877 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:08:53.346877 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:08:53.346877 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d" cmd=["/bin/sleep","15"]
Jul 09 19:08:53.347232 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.346897    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" message="" err=<
Jul 09 19:08:53.347232 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:08:53.347232 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:08:53.347232 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:08:53.347232 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:08:53.346992    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:08:53.347232 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:08:53.347232 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:08:53.347232 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" podUID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" containerName="main" containerID="cri-o://44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d"
Jul 09 19:08:53.347232 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.347044    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" podUID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" containerName="main" containerID="cri-o://44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d" gracePeriod=60
Jul 09 19:08:53.603096 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.603068    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:08:53.643889 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.643859    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tls-certs\") pod \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") "
Jul 09 19:08:53.644096 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.643945    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tokenizer-uds\") pod \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") "
Jul 09 19:08:53.644096 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.644019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kserve-provision-location\") pod \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") "
Jul 09 19:08:53.644096 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.644058    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8mh9\" (UniqueName: \"kubernetes.io/projected/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kube-api-access-j8mh9\") pod \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\" (UID: \"a09281d2-36ca-43fa-9d3d-0153ae7d5b09\") "
Jul 09 19:08:53.644263 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.644196    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a09281d2-36ca-43fa-9d3d-0153ae7d5b09" (UID: "a09281d2-36ca-43fa-9d3d-0153ae7d5b09"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:08:53.644346 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.644326    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:08:53.644696 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.644671    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a09281d2-36ca-43fa-9d3d-0153ae7d5b09" (UID: "a09281d2-36ca-43fa-9d3d-0153ae7d5b09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:08:53.646074 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.646052    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kube-api-access-j8mh9" (OuterVolumeSpecName: "kube-api-access-j8mh9") pod "a09281d2-36ca-43fa-9d3d-0153ae7d5b09" (UID: "a09281d2-36ca-43fa-9d3d-0153ae7d5b09"). InnerVolumeSpecName "kube-api-access-j8mh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:08:53.646170 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.646086    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a09281d2-36ca-43fa-9d3d-0153ae7d5b09" (UID: "a09281d2-36ca-43fa-9d3d-0153ae7d5b09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:08:53.744885 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.744845    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:08:53.744885 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.744880    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:08:53.744885 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:53.744889    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8mh9\" (UniqueName: \"kubernetes.io/projected/a09281d2-36ca-43fa-9d3d-0153ae7d5b09-kube-api-access-j8mh9\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:08:54.382521 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.382484    2565 generic.go:356] "Generic (PLEG): container finished" podID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" containerID="44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d" exitCode=0
Jul 09 19:08:54.383054 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.382538    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"
Jul 09 19:08:54.383054 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.382547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" event={"ID":"a09281d2-36ca-43fa-9d3d-0153ae7d5b09","Type":"ContainerDied","Data":"44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d"}
Jul 09 19:08:54.383054 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.382587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt" event={"ID":"a09281d2-36ca-43fa-9d3d-0153ae7d5b09","Type":"ContainerDied","Data":"faf72cded59fe1f18f5d4f4d202868220f499451660b1657ddd9ee8c204b0d22"}
Jul 09 19:08:54.383054 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.382608    2565 scope.go:117] "RemoveContainer" containerID="44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d"
Jul 09 19:08:54.391928 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.391879    2565 scope.go:117] "RemoveContainer" containerID="dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac"
Jul 09 19:08:54.399609 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.399581    2565 scope.go:117] "RemoveContainer" containerID="44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d"
Jul 09 19:08:54.399861 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:08:54.399842    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d\": container with ID starting with 44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d not found: ID does not exist" containerID="44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d"
Jul 09 19:08:54.399969 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.399874    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d"} err="failed to get container status \"44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d\": rpc error: code = NotFound desc = could not find container \"44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d\": container with ID starting with 44d0d0590387b9b2173f20c78f53801536ad9ed32b43763c97e07fc638107f0d not found: ID does not exist"
Jul 09 19:08:54.399969 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.399899    2565 scope.go:117] "RemoveContainer" containerID="dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac"
Jul 09 19:08:54.400235 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:08:54.400218    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac\": container with ID starting with dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac not found: ID does not exist" containerID="dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac"
Jul 09 19:08:54.400274 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.400241    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac"} err="failed to get container status \"dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac\": rpc error: code = NotFound desc = could not find container \"dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac\": container with ID starting with dcd42c9c8adb6c7da51a5752855f8ae8a5d7142550e97e34a26d491124e4f9ac not found: ID does not exist"
Jul 09 19:08:54.405660 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.405634    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"]
Jul 09 19:08:54.409775 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.409751    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-xxpmt"]
Jul 09 19:08:54.868084 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:08:54.868048    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" path="/var/lib/kubelet/pods/a09281d2-36ca-43fa-9d3d-0153ae7d5b09/volumes"
Jul 09 19:09:06.956677 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:06.956546    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main" containerID="cri-o://db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2" gracePeriod=45
Jul 09 19:09:07.725083 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.725054    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:09:07.780668 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.780580    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tmp-dir\") pod \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") "
Jul 09 19:09:07.780668 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.780618    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-model-cache\") pod \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") "
Jul 09 19:09:07.780668 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.780657    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tls-certs\") pod \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") "
Jul 09 19:09:07.780989 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.780702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kserve-provision-location\") pod \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") "
Jul 09 19:09:07.780989 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.780727    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-home\") pod \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") "
Jul 09 19:09:07.780989 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.780788    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcfwv\" (UniqueName: \"kubernetes.io/projected/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kube-api-access-mcfwv\") pod \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") "
Jul 09 19:09:07.780989 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.780812    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-dshm\") pod \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\" (UID: \"b96d804d-feb5-4c1f-a900-d16ff1afcd83\") "
Jul 09 19:09:07.780989 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.780946    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-model-cache" (OuterVolumeSpecName: "model-cache") pod "b96d804d-feb5-4c1f-a900-d16ff1afcd83" (UID: "b96d804d-feb5-4c1f-a900-d16ff1afcd83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:09:07.781299 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.781145    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:09:07.781492 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.781464    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-home" (OuterVolumeSpecName: "home") pod "b96d804d-feb5-4c1f-a900-d16ff1afcd83" (UID: "b96d804d-feb5-4c1f-a900-d16ff1afcd83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:09:07.782986 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.782958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b96d804d-feb5-4c1f-a900-d16ff1afcd83" (UID: "b96d804d-feb5-4c1f-a900-d16ff1afcd83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:09:07.783513 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.783483    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-dshm" (OuterVolumeSpecName: "dshm") pod "b96d804d-feb5-4c1f-a900-d16ff1afcd83" (UID: "b96d804d-feb5-4c1f-a900-d16ff1afcd83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:09:07.783606 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.783486    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kube-api-access-mcfwv" (OuterVolumeSpecName: "kube-api-access-mcfwv") pod "b96d804d-feb5-4c1f-a900-d16ff1afcd83" (UID: "b96d804d-feb5-4c1f-a900-d16ff1afcd83"). InnerVolumeSpecName "kube-api-access-mcfwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:09:07.793190 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.793166    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b96d804d-feb5-4c1f-a900-d16ff1afcd83" (UID: "b96d804d-feb5-4c1f-a900-d16ff1afcd83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:09:07.845384 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.845342    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b96d804d-feb5-4c1f-a900-d16ff1afcd83" (UID: "b96d804d-feb5-4c1f-a900-d16ff1afcd83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:09:07.882218 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.882179    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcfwv\" (UniqueName: \"kubernetes.io/projected/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kube-api-access-mcfwv\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:09:07.882218 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.882213    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:09:07.882218 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.882224    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:09:07.882446 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.882232    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96d804d-feb5-4c1f-a900-d16ff1afcd83-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:09:07.882446 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.882242    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:09:07.882446 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:07.882250    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96d804d-feb5-4c1f-a900-d16ff1afcd83-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:09:08.446625 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.446586    2565 generic.go:356] "Generic (PLEG): container finished" podID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerID="db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2" exitCode=0
Jul 09 19:09:08.447104 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.446643    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"
Jul 09 19:09:08.447104 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.446672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" event={"ID":"b96d804d-feb5-4c1f-a900-d16ff1afcd83","Type":"ContainerDied","Data":"db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2"}
Jul 09 19:09:08.447104 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.446726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n" event={"ID":"b96d804d-feb5-4c1f-a900-d16ff1afcd83","Type":"ContainerDied","Data":"e58cee8a8c4cd24da9e2444a5144c43084fc30f10051fb51992b7f7105f7e7a3"}
Jul 09 19:09:08.447104 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.446751    2565 scope.go:117] "RemoveContainer" containerID="db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2"
Jul 09 19:09:08.455816 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.455795    2565 scope.go:117] "RemoveContainer" containerID="5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29"
Jul 09 19:09:08.465165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.464281    2565 scope.go:117] "RemoveContainer" containerID="db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2"
Jul 09 19:09:08.465165 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:09:08.464683    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2\": container with ID starting with db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2 not found: ID does not exist" containerID="db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2"
Jul 09 19:09:08.465165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.464715    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2"} err="failed to get container status \"db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2\": rpc error: code = NotFound desc = could not find container \"db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2\": container with ID starting with db958186e7240f30154aa7815aa96eeb1172a55ee8a90a1e03c8534fd447d1a2 not found: ID does not exist"
Jul 09 19:09:08.465165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.464739    2565 scope.go:117] "RemoveContainer" containerID="5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29"
Jul 09 19:09:08.466025 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:09:08.465996    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29\": container with ID starting with 5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29 not found: ID does not exist" containerID="5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29"
Jul 09 19:09:08.466144 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.466031    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29"} err="failed to get container status \"5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29\": rpc error: code = NotFound desc = could not find container \"5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29\": container with ID starting with 5186162a1dc0b3f12fde1244248fe1d0c6294d3e06dd9a4b9e0ed1954006ff29 not found: ID does not exist"
Jul 09 19:09:08.470039 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.470017    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"]
Jul 09 19:09:08.473687 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.473657    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-pgk2n"]
Jul 09 19:09:08.868486 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:08.868408    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" path="/var/lib/kubelet/pods/b96d804d-feb5-4c1f-a900-d16ff1afcd83/volumes"
Jul 09 19:09:19.173397 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173362    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"]
Jul 09 19:09:19.173750 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173700    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" containerName="main"
Jul 09 19:09:19.173750 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173712    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" containerName="main"
Jul 09 19:09:19.173750 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173721    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="storage-initializer"
Jul 09 19:09:19.173750 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173727    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="storage-initializer"
Jul 09 19:09:19.173750 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173737    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" containerName="main"
Jul 09 19:09:19.173750 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173742    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" containerName="main"
Jul 09 19:09:19.173992 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173761    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" containerName="storage-initializer"
Jul 09 19:09:19.173992 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173768    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" containerName="storage-initializer"
Jul 09 19:09:19.173992 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main"
Jul 09 19:09:19.173992 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173791    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main"
Jul 09 19:09:19.173992 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173852    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96d804d-feb5-4c1f-a900-d16ff1afcd83" containerName="main"
Jul 09 19:09:19.173992 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173861    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="067fee28-f9e5-4ad6-acd2-9b2cb8f7a401" containerName="main"
Jul 09 19:09:19.173992 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.173868    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09281d2-36ca-43fa-9d3d-0153ae7d5b09" containerName="main"
Jul 09 19:09:19.178336 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.178316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.181221 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.181199    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 19:09:19.181365 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.181261    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 19:09:19.191376 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.191349    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"]
Jul 09 19:09:19.286038 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.285998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kserve-provision-location\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.286199 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.286055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-dshm\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.286199 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.286081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czpfb\" (UniqueName: \"kubernetes.io/projected/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kube-api-access-czpfb\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.286199 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.286117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tmp-dir\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.286342 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.286192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-model-cache\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.286342 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.286280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-home\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.286342 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.286312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tls-certs\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.379126 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.379092    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"]
Jul 09 19:09:19.382701 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.382675    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.385887 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.385864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-wsbzn\""
Jul 09 19:09:19.386792 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.386763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-home\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.386929 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.386814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tls-certs\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.386929 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.386853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kserve-provision-location\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.386929 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.386897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-dshm\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.387106 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.386940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czpfb\" (UniqueName: \"kubernetes.io/projected/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kube-api-access-czpfb\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.387106 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.386975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tmp-dir\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.387106 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.387007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-model-cache\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.387260 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.387200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-home\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.387462 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.387436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kserve-provision-location\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.387539 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.387525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tmp-dir\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.387667 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.387646    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-model-cache\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.389656 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.389629    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-dshm\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.389883 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.389862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tls-certs\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.397715 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.397687    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"]
Jul 09 19:09:19.399734 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.399710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czpfb\" (UniqueName: \"kubernetes.io/projected/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kube-api-access-czpfb\") pod \"stop-feature-test-kserve-5c675d88cb-v8rvq\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.487523 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.487486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.487689 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.487551    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.487689 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.487674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddxj\" (UniqueName: \"kubernetes.io/projected/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kube-api-access-rddxj\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.487818 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.487719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.488361 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.488341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:19.588734 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.588697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rddxj\" (UniqueName: \"kubernetes.io/projected/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kube-api-access-rddxj\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.588945 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.588750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.588945 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.588795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.588945 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.588855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.589269 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.589245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.589347 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.589247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.591500 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.591447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.597840 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.597811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddxj\" (UniqueName: \"kubernetes.io/projected/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kube-api-access-rddxj\") pod \"stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.629619 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.629593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"]
Jul 09 19:09:19.631701 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:09:19.631671    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7de3d9_bd99_46e7_9d5d_efe2cf2044b2.slice/crio-c200028c6e639f59f4ac7385196040ff9d577541f4e29880162ae3171531d17f WatchSource:0}: Error finding container c200028c6e639f59f4ac7385196040ff9d577541f4e29880162ae3171531d17f: Status 404 returned error can't find the container with id c200028c6e639f59f4ac7385196040ff9d577541f4e29880162ae3171531d17f
Jul 09 19:09:19.702134 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.702099    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:19.838034 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:19.838003    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"]
Jul 09 19:09:19.839619 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:09:19.839587    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791eb984_b8e0_49cd_8ee7_00849ef8adc1.slice/crio-b3ec600f4aa44f46a09c04fd738192b4f5ad3b13a8077b7c606f81619d688fe2 WatchSource:0}: Error finding container b3ec600f4aa44f46a09c04fd738192b4f5ad3b13a8077b7c606f81619d688fe2: Status 404 returned error can't find the container with id b3ec600f4aa44f46a09c04fd738192b4f5ad3b13a8077b7c606f81619d688fe2
Jul 09 19:09:20.502899 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:20.502858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" event={"ID":"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2","Type":"ContainerStarted","Data":"299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403"}
Jul 09 19:09:20.503376 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:20.502939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" event={"ID":"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2","Type":"ContainerStarted","Data":"c200028c6e639f59f4ac7385196040ff9d577541f4e29880162ae3171531d17f"}
Jul 09 19:09:20.504468 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:20.504444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" event={"ID":"791eb984-b8e0-49cd-8ee7-00849ef8adc1","Type":"ContainerStarted","Data":"12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa"}
Jul 09 19:09:20.504615 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:20.504481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" event={"ID":"791eb984-b8e0-49cd-8ee7-00849ef8adc1","Type":"ContainerStarted","Data":"b3ec600f4aa44f46a09c04fd738192b4f5ad3b13a8077b7c606f81619d688fe2"}
Jul 09 19:09:21.519393 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:21.519359    2565 generic.go:356] "Generic (PLEG): container finished" podID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" containerID="12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa" exitCode=0
Jul 09 19:09:21.519956 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:21.519529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" event={"ID":"791eb984-b8e0-49cd-8ee7-00849ef8adc1","Type":"ContainerDied","Data":"12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa"}
Jul 09 19:09:22.527476 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:22.527426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" event={"ID":"791eb984-b8e0-49cd-8ee7-00849ef8adc1","Type":"ContainerStarted","Data":"909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da"}
Jul 09 19:09:22.528024 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:22.527516    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:22.552753 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:22.552691    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" podStartSLOduration=2.885454149 podStartE2EDuration="3.552675535s" totalImagesPullingTime="0s" totalInitContainerRuntime="667.221386ms" isStatefulPod=false podCreationTimestamp="2026-07-09 19:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:09:22.551419668 +0000 UTC m=+3882.269682432" watchObservedRunningTime="2026-07-09 19:09:22.552675535 +0000 UTC m=+3882.270938299"
Jul 09 19:09:23.533413 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:23.533376    2565 generic.go:356] "Generic (PLEG): container finished" podID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerID="299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403" exitCode=0
Jul 09 19:09:23.533808 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:23.533450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" event={"ID":"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2","Type":"ContainerDied","Data":"299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403"}
Jul 09 19:09:24.541051 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:24.541016    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" event={"ID":"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2","Type":"ContainerStarted","Data":"b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133"}
Jul 09 19:09:24.567972 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:24.567897    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podStartSLOduration=2.37345337 podStartE2EDuration="5.567878295s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.194424925s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:09:24.565823242 +0000 UTC m=+3884.284086006" watchObservedRunningTime="2026-07-09 19:09:24.567878295 +0000 UTC m=+3884.286141060"
Jul 09 19:09:25.489422 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:25.489387    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:29.489113 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:29.489073    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:09:29.490927 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:29.490885    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:09:39.489073 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:39.489030    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:09:41.007986 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:41.007840    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:09:41.015703 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:41.015678    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:09:49.488852 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:49.488799    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:09:53.535717 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:53.535687    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:09:59.489261 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:09:59.489209    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:10:09.489111 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:10:09.489070    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:10:19.489427 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:10:19.489381    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:10:29.489133 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:10:29.489087    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:10:39.489461 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:10:39.489365    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:10:49.489620 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:10:49.489578    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:10:59.489821 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:10:59.489762    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:11:09.489162 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:11:09.489115    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:11:19.499327 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:11:19.499294    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:11:19.507502 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:11:19.507471    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:12:22.932631 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:22.932594    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"]
Jul 09 19:12:22.935970 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:22.935950    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:22.938886 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:22.938865    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 09 19:12:22.939787 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:22.939769    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-z95d9\""
Jul 09 19:12:22.947354 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:22.947326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"]
Jul 09 19:12:23.055184 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.055141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dn5p\" (UniqueName: \"kubernetes.io/projected/219b5055-da02-4a42-bb94-c1165b28d6fa-kube-api-access-9dn5p\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.055371 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.055212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/219b5055-da02-4a42-bb94-c1165b28d6fa-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.055371 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.055348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219b5055-da02-4a42-bb94-c1165b28d6fa-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.156741 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.156689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219b5055-da02-4a42-bb94-c1165b28d6fa-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.156926 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.156768    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dn5p\" (UniqueName: \"kubernetes.io/projected/219b5055-da02-4a42-bb94-c1165b28d6fa-kube-api-access-9dn5p\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.156926 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.156817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/219b5055-da02-4a42-bb94-c1165b28d6fa-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.157303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.157275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/219b5055-da02-4a42-bb94-c1165b28d6fa-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.159340 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.159319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219b5055-da02-4a42-bb94-c1165b28d6fa-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.166473 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.166449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dn5p\" (UniqueName: \"kubernetes.io/projected/219b5055-da02-4a42-bb94-c1165b28d6fa-kube-api-access-9dn5p\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.246480 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.246444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:23.385349 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.385323    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"]
Jul 09 19:12:23.387936 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:12:23.387882    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219b5055_da02_4a42_bb94_c1165b28d6fa.slice/crio-a14157887c1fedbbe88e83d53034826e0948deb3d33953f0787f70a1678b682a WatchSource:0}: Error finding container a14157887c1fedbbe88e83d53034826e0948deb3d33953f0787f70a1678b682a: Status 404 returned error can't find the container with id a14157887c1fedbbe88e83d53034826e0948deb3d33953f0787f70a1678b682a
Jul 09 19:12:23.405402 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:23.405373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j" event={"ID":"219b5055-da02-4a42-bb94-c1165b28d6fa","Type":"ContainerStarted","Data":"a14157887c1fedbbe88e83d53034826e0948deb3d33953f0787f70a1678b682a"}
Jul 09 19:12:24.412506 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:24.412471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j" event={"ID":"219b5055-da02-4a42-bb94-c1165b28d6fa","Type":"ContainerStarted","Data":"e098f696dc7971b2ec6b0b00bb8d16cd02dc54bd4f6db8891ea6ed9dc48cf378"}
Jul 09 19:12:24.412917 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:24.412530    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:12:24.431734 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:24.431678    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j" podStartSLOduration=2.431662208 podStartE2EDuration="2.431662208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:12:24.430216049 +0000 UTC m=+4064.148478813" watchObservedRunningTime="2026-07-09 19:12:24.431662208 +0000 UTC m=+4064.149924972"
Jul 09 19:12:37.113430 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.113393    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"]
Jul 09 19:12:37.115538 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.115510    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"]
Jul 09 19:12:37.123037 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:12:37.123000    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:12:37.123037 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:12:37.123037 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:12:37.123037 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da" cmd=["/bin/sleep","15"]
Jul 09 19:12:37.123354 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.123050    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" message="" err=<
Jul 09 19:12:37.123354 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:12:37.123354 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:12:37.123354 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:12:37.123354 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:12:37.123064    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:12:37.123354 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:12:37.123354 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:12:37.123354 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" podUID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" containerName="main" containerID="cri-o://909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da"
Jul 09 19:12:37.123354 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.123119    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" podUID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" containerName="main" containerID="cri-o://909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da" gracePeriod=60
Jul 09 19:12:37.375808 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.375742    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:12:37.472406 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.472372    2565 generic.go:356] "Generic (PLEG): container finished" podID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" containerID="909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da" exitCode=0
Jul 09 19:12:37.472607 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.472428    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"
Jul 09 19:12:37.472607 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.472452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" event={"ID":"791eb984-b8e0-49cd-8ee7-00849ef8adc1","Type":"ContainerDied","Data":"909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da"}
Jul 09 19:12:37.472607 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.472510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm" event={"ID":"791eb984-b8e0-49cd-8ee7-00849ef8adc1","Type":"ContainerDied","Data":"b3ec600f4aa44f46a09c04fd738192b4f5ad3b13a8077b7c606f81619d688fe2"}
Jul 09 19:12:37.472607 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.472532    2565 scope.go:117] "RemoveContainer" containerID="909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da"
Jul 09 19:12:37.481353 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.481334    2565 scope.go:117] "RemoveContainer" containerID="12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa"
Jul 09 19:12:37.488494 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.488471    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kserve-provision-location\") pod \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") "
Jul 09 19:12:37.488608 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.488541    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tls-certs\") pod \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") "
Jul 09 19:12:37.488608 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.488574    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rddxj\" (UniqueName: \"kubernetes.io/projected/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kube-api-access-rddxj\") pod \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") "
Jul 09 19:12:37.488714 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.488634    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tokenizer-uds\") pod \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\" (UID: \"791eb984-b8e0-49cd-8ee7-00849ef8adc1\") "
Jul 09 19:12:37.489028 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.488997    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "791eb984-b8e0-49cd-8ee7-00849ef8adc1" (UID: "791eb984-b8e0-49cd-8ee7-00849ef8adc1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:12:37.489438 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.489410    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "791eb984-b8e0-49cd-8ee7-00849ef8adc1" (UID: "791eb984-b8e0-49cd-8ee7-00849ef8adc1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:12:37.489529 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.489422    2565 scope.go:117] "RemoveContainer" containerID="909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da"
Jul 09 19:12:37.489747 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:12:37.489730    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da\": container with ID starting with 909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da not found: ID does not exist" containerID="909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da"
Jul 09 19:12:37.489814 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.489754    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da"} err="failed to get container status \"909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da\": rpc error: code = NotFound desc = could not find container \"909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da\": container with ID starting with 909a500b4e545ecedd44a83862e04b1cee87418b4f4c77e1600f2642ad3f80da not found: ID does not exist"
Jul 09 19:12:37.489814 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.489771    2565 scope.go:117] "RemoveContainer" containerID="12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa"
Jul 09 19:12:37.497141 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:12:37.496978    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa\": container with ID starting with 12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa not found: ID does not exist" containerID="12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa"
Jul 09 19:12:37.497141 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.497020    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa"} err="failed to get container status \"12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa\": rpc error: code = NotFound desc = could not find container \"12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa\": container with ID starting with 12ffd7c302391e95934c91801fdea50fa4610f7a647facb3d5fbd9339caa95aa not found: ID does not exist"
Jul 09 19:12:37.497698 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.497676    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "791eb984-b8e0-49cd-8ee7-00849ef8adc1" (UID: "791eb984-b8e0-49cd-8ee7-00849ef8adc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:12:37.497823 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.497692    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kube-api-access-rddxj" (OuterVolumeSpecName: "kube-api-access-rddxj") pod "791eb984-b8e0-49cd-8ee7-00849ef8adc1" (UID: "791eb984-b8e0-49cd-8ee7-00849ef8adc1"). InnerVolumeSpecName "kube-api-access-rddxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:12:37.589595 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.589551    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rddxj\" (UniqueName: \"kubernetes.io/projected/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kube-api-access-rddxj\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:37.589595 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.589587    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:37.589595 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.589598    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791eb984-b8e0-49cd-8ee7-00849ef8adc1-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:37.589856 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.589608    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791eb984-b8e0-49cd-8ee7-00849ef8adc1-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:37.796391 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.796355    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"]
Jul 09 19:12:37.800973 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:37.800942    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-79c8c67dd9-5kfwm"]
Jul 09 19:12:38.868189 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:38.868157    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" path="/var/lib/kubelet/pods/791eb984-b8e0-49cd-8ee7-00849ef8adc1/volumes"
Jul 09 19:12:52.140897 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:52.140822    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" containerID="cri-o://b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133" gracePeriod=45
Jul 09 19:12:52.500670 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:52.500626    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 09 19:12:52.900981 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:52.900955    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:12:53.017956 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.017855    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czpfb\" (UniqueName: \"kubernetes.io/projected/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kube-api-access-czpfb\") pod \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") "
Jul 09 19:12:53.017956 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.017924    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kserve-provision-location\") pod \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") "
Jul 09 19:12:53.017956 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.017947    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tls-certs\") pod \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") "
Jul 09 19:12:53.018284 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.017985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tmp-dir\") pod \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") "
Jul 09 19:12:53.018284 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.018042    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-model-cache\") pod \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") "
Jul 09 19:12:53.018284 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.018080    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-dshm\") pod \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") "
Jul 09 19:12:53.018284 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.018103    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-home\") pod \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\" (UID: \"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2\") "
Jul 09 19:12:53.018499 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.018403    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" (UID: "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:12:53.018876 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.018845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-home" (OuterVolumeSpecName: "home") pod "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" (UID: "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:12:53.020595 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.020567    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kube-api-access-czpfb" (OuterVolumeSpecName: "kube-api-access-czpfb") pod "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" (UID: "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2"). InnerVolumeSpecName "kube-api-access-czpfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:12:53.020595 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.020567    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-dshm" (OuterVolumeSpecName: "dshm") pod "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" (UID: "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:12:53.020725 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.020657    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" (UID: "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:12:53.029970 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.029940    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" (UID: "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:12:53.084760 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.084714    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" (UID: "aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:12:53.119365 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.119326    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:53.119365 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.119359    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:53.119365 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.119369    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czpfb\" (UniqueName: \"kubernetes.io/projected/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kube-api-access-czpfb\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:53.119604 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.119380    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:53.119604 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.119393    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:53.119604 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.119426    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:53.119604 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.119434    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:12:53.546080 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.546044    2565 generic.go:356] "Generic (PLEG): container finished" podID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerID="b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133" exitCode=0
Jul 09 19:12:53.546464 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.546096    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"
Jul 09 19:12:53.546464 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.546133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" event={"ID":"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2","Type":"ContainerDied","Data":"b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133"}
Jul 09 19:12:53.546464 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.546188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq" event={"ID":"aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2","Type":"ContainerDied","Data":"c200028c6e639f59f4ac7385196040ff9d577541f4e29880162ae3171531d17f"}
Jul 09 19:12:53.546464 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.546211    2565 scope.go:117] "RemoveContainer" containerID="b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133"
Jul 09 19:12:53.554924 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.554880    2565 scope.go:117] "RemoveContainer" containerID="299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403"
Jul 09 19:12:53.563203 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.563187    2565 scope.go:117] "RemoveContainer" containerID="b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133"
Jul 09 19:12:53.563446 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:12:53.563427    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133\": container with ID starting with b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133 not found: ID does not exist" containerID="b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133"
Jul 09 19:12:53.563509 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.563451    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133"} err="failed to get container status \"b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133\": rpc error: code = NotFound desc = could not find container \"b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133\": container with ID starting with b8fdab9f207ce00b51ab27d4ae7b30d67f876bc055976c2dc73a928daf81b133 not found: ID does not exist"
Jul 09 19:12:53.563509 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.563466    2565 scope.go:117] "RemoveContainer" containerID="299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403"
Jul 09 19:12:53.563708 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:12:53.563693    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403\": container with ID starting with 299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403 not found: ID does not exist" containerID="299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403"
Jul 09 19:12:53.563756 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.563710    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403"} err="failed to get container status \"299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403\": rpc error: code = NotFound desc = could not find container \"299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403\": container with ID starting with 299ba9e7c0f8ac03f8c7ce532dc89c921d175d030ce810fc74a722370ab22403 not found: ID does not exist"
Jul 09 19:12:53.570410 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.570383    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"]
Jul 09 19:12:53.574426 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:53.574399    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5c675d88cb-v8rvq"]
Jul 09 19:12:54.867725 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:54.867693    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" path="/var/lib/kubelet/pods/aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2/volumes"
Jul 09 19:12:55.418484 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:12:55.418454    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:13:33.375015 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.374940    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"]
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375304    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="storage-initializer"
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375318    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="storage-initializer"
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375327    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" containerName="storage-initializer"
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375333    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" containerName="storage-initializer"
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375350    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main"
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375355    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main"
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375369    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" containerName="main"
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375374    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" containerName="main"
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375425    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7de3d9-bd99-46e7-9d5d-efe2cf2044b2" containerName="main"
Jul 09 19:13:33.375482 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.375433    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="791eb984-b8e0-49cd-8ee7-00849ef8adc1" containerName="main"
Jul 09 19:13:33.378330 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.378313    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.381404 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.381375    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 09 19:13:33.381511 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.381404    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 19:13:33.381511 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.381451    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 19:13:33.394155 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.394128    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"]
Jul 09 19:13:33.475614 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.475584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-model-cache\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.475804 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.475626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.475804 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.475653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-dshm\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.475804 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.475672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-home\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.475804 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.475707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.475804 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.475775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tls-certs\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.476016 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.475810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfqbh\" (UniqueName: \"kubernetes.io/projected/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kube-api-access-sfqbh\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.476016 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.475836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tmp-dir\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576229 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-home\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576424 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576424 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tls-certs\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576424 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfqbh\" (UniqueName: \"kubernetes.io/projected/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kube-api-access-sfqbh\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576424 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tmp-dir\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576424 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-model-cache\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576717 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576717 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-dshm\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576717 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-home\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576882 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576747    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-model-cache\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.576882 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576771    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.577136 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.576973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tmp-dir\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.577466 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.577419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.583930 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.579423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tls-certs\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.584763 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.584738    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-dshm\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.585822 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.585792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfqbh\" (UniqueName: \"kubernetes.io/projected/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kube-api-access-sfqbh\") pod \"lora-single-adapter-test-kserve-5d5f5494d5-d5r4t\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.688397 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.688302    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:33.829581 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:33.829551    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"]
Jul 09 19:13:33.831643 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:13:33.831614    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6cebbf_b9ce_47e2_8cfb_71a03e429dca.slice/crio-8a787b6aa8f69429ee1a16e5625f7366e0db4b3eed79b73816dcdc283a8edbd1 WatchSource:0}: Error finding container 8a787b6aa8f69429ee1a16e5625f7366e0db4b3eed79b73816dcdc283a8edbd1: Status 404 returned error can't find the container with id 8a787b6aa8f69429ee1a16e5625f7366e0db4b3eed79b73816dcdc283a8edbd1
Jul 09 19:13:34.737081 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:34.737044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" event={"ID":"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca","Type":"ContainerStarted","Data":"505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d"}
Jul 09 19:13:34.737529 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:34.737108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" event={"ID":"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca","Type":"ContainerStarted","Data":"8a787b6aa8f69429ee1a16e5625f7366e0db4b3eed79b73816dcdc283a8edbd1"}
Jul 09 19:13:37.751737 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:37.751646    2565 generic.go:356] "Generic (PLEG): container finished" podID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerID="505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d" exitCode=0
Jul 09 19:13:37.751737 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:37.751724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" event={"ID":"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca","Type":"ContainerDied","Data":"505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d"}
Jul 09 19:13:38.759330 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:38.759292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" event={"ID":"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca","Type":"ContainerStarted","Data":"ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70"}
Jul 09 19:13:38.786739 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:38.786690    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podStartSLOduration=2.189098731 podStartE2EDuration="5.786671711s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.59757298s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:13:38.783867992 +0000 UTC m=+4138.502130792" watchObservedRunningTime="2026-07-09 19:13:38.786671711 +0000 UTC m=+4138.504934474"
Jul 09 19:13:39.689043 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:39.689003    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:43.689427 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:43.689384    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:13:43.690988 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:43.690958    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 09 19:13:53.689106 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:13:53.689061    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 09 19:14:00.738638 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:00.738599    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"]
Jul 09 19:14:00.748817 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:14:00.748778    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:14:00.748817 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:14:00.748817 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:14:00.748817 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="e098f696dc7971b2ec6b0b00bb8d16cd02dc54bd4f6db8891ea6ed9dc48cf378" cmd=["/bin/sleep","15"]
Jul 09 19:14:00.749215 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:00.748862    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j" message="" err=<
Jul 09 19:14:00.749215 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:14:00.749215 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:14:00.749215 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:14:00.749215 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:14:00.748880    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:14:00.749215 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:14:00.749215 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:14:00.749215 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j" podUID="219b5055-da02-4a42-bb94-c1165b28d6fa" containerName="main" containerID="cri-o://e098f696dc7971b2ec6b0b00bb8d16cd02dc54bd4f6db8891ea6ed9dc48cf378"
Jul 09 19:14:00.749215 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:00.748941    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j" podUID="219b5055-da02-4a42-bb94-c1165b28d6fa" containerName="main" containerID="cri-o://e098f696dc7971b2ec6b0b00bb8d16cd02dc54bd4f6db8891ea6ed9dc48cf378" gracePeriod=60
Jul 09 19:14:00.871946 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:00.871729    2565 generic.go:356] "Generic (PLEG): container finished" podID="219b5055-da02-4a42-bb94-c1165b28d6fa" containerID="e098f696dc7971b2ec6b0b00bb8d16cd02dc54bd4f6db8891ea6ed9dc48cf378" exitCode=0
Jul 09 19:14:00.876441 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:00.876410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j" event={"ID":"219b5055-da02-4a42-bb94-c1165b28d6fa","Type":"ContainerDied","Data":"e098f696dc7971b2ec6b0b00bb8d16cd02dc54bd4f6db8891ea6ed9dc48cf378"}
Jul 09 19:14:01.020105 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.020082    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:14:01.133512 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.133475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dn5p\" (UniqueName: \"kubernetes.io/projected/219b5055-da02-4a42-bb94-c1165b28d6fa-kube-api-access-9dn5p\") pod \"219b5055-da02-4a42-bb94-c1165b28d6fa\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") "
Jul 09 19:14:01.133696 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.133581    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/219b5055-da02-4a42-bb94-c1165b28d6fa-tokenizer-uds\") pod \"219b5055-da02-4a42-bb94-c1165b28d6fa\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") "
Jul 09 19:14:01.133696 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.133626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219b5055-da02-4a42-bb94-c1165b28d6fa-tls-certs\") pod \"219b5055-da02-4a42-bb94-c1165b28d6fa\" (UID: \"219b5055-da02-4a42-bb94-c1165b28d6fa\") "
Jul 09 19:14:01.133897 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.133867    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219b5055-da02-4a42-bb94-c1165b28d6fa-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "219b5055-da02-4a42-bb94-c1165b28d6fa" (UID: "219b5055-da02-4a42-bb94-c1165b28d6fa"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:14:01.135814 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.135780    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219b5055-da02-4a42-bb94-c1165b28d6fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "219b5055-da02-4a42-bb94-c1165b28d6fa" (UID: "219b5055-da02-4a42-bb94-c1165b28d6fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:14:01.135946 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.135863    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219b5055-da02-4a42-bb94-c1165b28d6fa-kube-api-access-9dn5p" (OuterVolumeSpecName: "kube-api-access-9dn5p") pod "219b5055-da02-4a42-bb94-c1165b28d6fa" (UID: "219b5055-da02-4a42-bb94-c1165b28d6fa"). InnerVolumeSpecName "kube-api-access-9dn5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:14:01.234431 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.234390    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/219b5055-da02-4a42-bb94-c1165b28d6fa-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:14:01.234431 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.234423    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219b5055-da02-4a42-bb94-c1165b28d6fa-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:14:01.234431 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.234435    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dn5p\" (UniqueName: \"kubernetes.io/projected/219b5055-da02-4a42-bb94-c1165b28d6fa-kube-api-access-9dn5p\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:14:01.878065 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.878031    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"
Jul 09 19:14:01.878065 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.878057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j" event={"ID":"219b5055-da02-4a42-bb94-c1165b28d6fa","Type":"ContainerDied","Data":"a14157887c1fedbbe88e83d53034826e0948deb3d33953f0787f70a1678b682a"}
Jul 09 19:14:01.878570 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.878115    2565 scope.go:117] "RemoveContainer" containerID="e098f696dc7971b2ec6b0b00bb8d16cd02dc54bd4f6db8891ea6ed9dc48cf378"
Jul 09 19:14:01.901305 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.901271    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"]
Jul 09 19:14:01.905300 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:01.905266    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dp9d2j"]
Jul 09 19:14:02.869053 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:02.869020    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="219b5055-da02-4a42-bb94-c1165b28d6fa" path="/var/lib/kubelet/pods/219b5055-da02-4a42-bb94-c1165b28d6fa/volumes"
Jul 09 19:14:03.689675 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:03.689632    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 09 19:14:13.689129 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:13.689086    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 09 19:14:23.689670 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:23.689628    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 09 19:14:33.689382 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:33.689324    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 09 19:14:41.039011 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:41.038882    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:14:41.048483 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:41.048450    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:14:43.689371 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:43.689322    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 09 19:14:53.699423 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:53.699389    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:14:53.707640 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:14:53.707613    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:15:01.995418 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:01.995320    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"]
Jul 09 19:15:01.995879 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:01.995761    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219b5055-da02-4a42-bb94-c1165b28d6fa" containerName="main"
Jul 09 19:15:01.995879 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:01.995775    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="219b5055-da02-4a42-bb94-c1165b28d6fa" containerName="main"
Jul 09 19:15:01.995879 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:01.995864    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="219b5055-da02-4a42-bb94-c1165b28d6fa" containerName="main"
Jul 09 19:15:01.998071 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:01.998055    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.002175 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.001281    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 09 19:15:02.002933 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.002422    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-epp-sa-dockercfg-q9bgc\""
Jul 09 19:15:02.010084 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.010050    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"]
Jul 09 19:15:02.086735 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.086694    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.086735 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.086736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.087033 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.086761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.087033 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.086807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.087033 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.086933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zgt\" (UniqueName: \"kubernetes.io/projected/436bcaf4-163b-4270-aa0f-8f066d2cf86c-kube-api-access-x8zgt\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.187312 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.187278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.187312 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.187317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.187528 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.187408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zgt\" (UniqueName: \"kubernetes.io/projected/436bcaf4-163b-4270-aa0f-8f066d2cf86c-kube-api-access-x8zgt\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.187528 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.187447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.187528 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.187476    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.187797 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.187773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.187839 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.187786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.187880 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.187840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.190106 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.190081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.198223 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.198199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zgt\" (UniqueName: \"kubernetes.io/projected/436bcaf4-163b-4270-aa0f-8f066d2cf86c-kube-api-access-x8zgt\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5678mw66\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.314124 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.314020    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:02.452695 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:02.452661    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"]
Jul 09 19:15:02.454140 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:15:02.454110    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436bcaf4_163b_4270_aa0f_8f066d2cf86c.slice/crio-efe84130db3ed3cb0bcb87c29fdd0aa9e306b14d38eb7d458716a558062046f8 WatchSource:0}: Error finding container efe84130db3ed3cb0bcb87c29fdd0aa9e306b14d38eb7d458716a558062046f8: Status 404 returned error can't find the container with id efe84130db3ed3cb0bcb87c29fdd0aa9e306b14d38eb7d458716a558062046f8
Jul 09 19:15:03.182747 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:03.182707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66" event={"ID":"436bcaf4-163b-4270-aa0f-8f066d2cf86c","Type":"ContainerStarted","Data":"89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32"}
Jul 09 19:15:03.183161 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:03.182764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66" event={"ID":"436bcaf4-163b-4270-aa0f-8f066d2cf86c","Type":"ContainerStarted","Data":"efe84130db3ed3cb0bcb87c29fdd0aa9e306b14d38eb7d458716a558062046f8"}
Jul 09 19:15:03.183161 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:03.182793    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:15:03.206495 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:03.206443    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66" podStartSLOduration=2.206426916 podStartE2EDuration="2.206426916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:15:03.205076949 +0000 UTC m=+4222.923339714" watchObservedRunningTime="2026-07-09 19:15:03.206426916 +0000 UTC m=+4222.924689679"
Jul 09 19:15:34.189676 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:15:34.189645    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:16:05.398067 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.398020    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"]
Jul 09 19:16:05.407540 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:16:05.407497    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:16:05.407540 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:16:05.407540 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:16:05.407540 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32" cmd=["/bin/sleep","15"]
Jul 09 19:16:05.407896 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.407559    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66" message="" err=<
Jul 09 19:16:05.407896 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:16:05.407896 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:16:05.407896 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:16:05.407896 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:16:05.407576    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:16:05.407896 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:16:05.407896 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:16:05.407896 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66" podUID="436bcaf4-163b-4270-aa0f-8f066d2cf86c" containerName="main" containerID="cri-o://89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32"
Jul 09 19:16:05.407896 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.407642    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66" podUID="436bcaf4-163b-4270-aa0f-8f066d2cf86c" containerName="main" containerID="cri-o://89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32" gracePeriod=60
Jul 09 19:16:05.657604 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.657535    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:16:05.749660 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.749630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-cachi2-cache\") pod \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") "
Jul 09 19:16:05.749660 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.749663    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-uds\") pod \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") "
Jul 09 19:16:05.749915 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.749695    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-cache\") pod \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") "
Jul 09 19:16:05.749915 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.749745    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tls-certs\") pod \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") "
Jul 09 19:16:05.749915 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.749794    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8zgt\" (UniqueName: \"kubernetes.io/projected/436bcaf4-163b-4270-aa0f-8f066d2cf86c-kube-api-access-x8zgt\") pod \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\" (UID: \"436bcaf4-163b-4270-aa0f-8f066d2cf86c\") "
Jul 09 19:16:05.750092 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.750000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "436bcaf4-163b-4270-aa0f-8f066d2cf86c" (UID: "436bcaf4-163b-4270-aa0f-8f066d2cf86c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:16:05.750092 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.750008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "436bcaf4-163b-4270-aa0f-8f066d2cf86c" (UID: "436bcaf4-163b-4270-aa0f-8f066d2cf86c"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:16:05.750092 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.750049    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "436bcaf4-163b-4270-aa0f-8f066d2cf86c" (UID: "436bcaf4-163b-4270-aa0f-8f066d2cf86c"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:16:05.751990 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.751958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436bcaf4-163b-4270-aa0f-8f066d2cf86c-kube-api-access-x8zgt" (OuterVolumeSpecName: "kube-api-access-x8zgt") pod "436bcaf4-163b-4270-aa0f-8f066d2cf86c" (UID: "436bcaf4-163b-4270-aa0f-8f066d2cf86c"). InnerVolumeSpecName "kube-api-access-x8zgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:16:05.751990 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.751968    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "436bcaf4-163b-4270-aa0f-8f066d2cf86c" (UID: "436bcaf4-163b-4270-aa0f-8f066d2cf86c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:16:05.851242 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.851202    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zgt\" (UniqueName: \"kubernetes.io/projected/436bcaf4-163b-4270-aa0f-8f066d2cf86c-kube-api-access-x8zgt\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:05.851242 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.851237    2565 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-cachi2-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:05.851242 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.851246    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:05.851242 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.851254    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tokenizer-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:05.851513 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:05.851262    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/436bcaf4-163b-4270-aa0f-8f066d2cf86c-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:06.484609 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.484574    2565 generic.go:356] "Generic (PLEG): container finished" podID="436bcaf4-163b-4270-aa0f-8f066d2cf86c" containerID="89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32" exitCode=0
Jul 09 19:16:06.485123 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.484618    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"
Jul 09 19:16:06.485123 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.484647    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66" event={"ID":"436bcaf4-163b-4270-aa0f-8f066d2cf86c","Type":"ContainerDied","Data":"89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32"}
Jul 09 19:16:06.485123 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.484697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66" event={"ID":"436bcaf4-163b-4270-aa0f-8f066d2cf86c","Type":"ContainerDied","Data":"efe84130db3ed3cb0bcb87c29fdd0aa9e306b14d38eb7d458716a558062046f8"}
Jul 09 19:16:06.485123 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.484725    2565 scope.go:117] "RemoveContainer" containerID="89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32"
Jul 09 19:16:06.494326 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.494306    2565 scope.go:117] "RemoveContainer" containerID="89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32"
Jul 09 19:16:06.494604 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:16:06.494584    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32\": container with ID starting with 89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32 not found: ID does not exist" containerID="89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32"
Jul 09 19:16:06.494671 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.494614    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32"} err="failed to get container status \"89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32\": rpc error: code = NotFound desc = could not find container \"89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32\": container with ID starting with 89e604e1b206f08d476cebf69ca3b873f4b6eb0f1c8b00dce6df81cf1e48df32 not found: ID does not exist"
Jul 09 19:16:06.511833 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.511801    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"]
Jul 09 19:16:06.520367 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.520337    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5678mw66"]
Jul 09 19:16:06.869892 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:06.869764    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="436bcaf4-163b-4270-aa0f-8f066d2cf86c" path="/var/lib/kubelet/pods/436bcaf4-163b-4270-aa0f-8f066d2cf86c/volumes"
Jul 09 19:16:22.462959 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:22.462922    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"]
Jul 09 19:16:37.486004 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:37.485897    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main" containerID="cri-o://ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70" gracePeriod=45
Jul 09 19:16:37.701134 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:37.701083    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 09 19:16:38.144453 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.144426    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:16:38.256332 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256301    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-dshm\") pod \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") "
Jul 09 19:16:38.256549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kserve-provision-location\") pod \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") "
Jul 09 19:16:38.256549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256388    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tls-certs\") pod \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") "
Jul 09 19:16:38.256549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-model-cache\") pod \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") "
Jul 09 19:16:38.256549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256437    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-home\") pod \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") "
Jul 09 19:16:38.256549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfqbh\" (UniqueName: \"kubernetes.io/projected/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kube-api-access-sfqbh\") pod \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") "
Jul 09 19:16:38.256843 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256563    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-cabundle-cert\") pod \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") "
Jul 09 19:16:38.256843 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tmp-dir\") pod \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\" (UID: \"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca\") "
Jul 09 19:16:38.256843 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256704    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" (UID: "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:16:38.257039 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256936    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:38.257039 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.256945    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-home" (OuterVolumeSpecName: "home") pod "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" (UID: "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:16:38.257271 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.257248    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" (UID: "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:16:38.258665 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.258640    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-dshm" (OuterVolumeSpecName: "dshm") pod "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" (UID: "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:16:38.258665 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.258645    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" (UID: "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:16:38.259169 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.259146    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kube-api-access-sfqbh" (OuterVolumeSpecName: "kube-api-access-sfqbh") pod "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" (UID: "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca"). InnerVolumeSpecName "kube-api-access-sfqbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:16:38.267784 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.267755    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" (UID: "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:16:38.323349 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.323313    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" (UID: "7a6cebbf-b9ce-47e2-8cfb-71a03e429dca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:16:38.358124 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.358094    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-cabundle-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:38.358124 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.358121    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:38.358294 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.358131    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:38.358294 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.358141    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:38.358294 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.358151    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:38.358294 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.358159    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:38.358294 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.358169    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfqbh\" (UniqueName: \"kubernetes.io/projected/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca-kube-api-access-sfqbh\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:16:38.650274 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.650181    2565 generic.go:356] "Generic (PLEG): container finished" podID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerID="ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70" exitCode=0
Jul 09 19:16:38.650274 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.650238    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"
Jul 09 19:16:38.650722 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.650270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" event={"ID":"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca","Type":"ContainerDied","Data":"ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70"}
Jul 09 19:16:38.650722 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.650331    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t" event={"ID":"7a6cebbf-b9ce-47e2-8cfb-71a03e429dca","Type":"ContainerDied","Data":"8a787b6aa8f69429ee1a16e5625f7366e0db4b3eed79b73816dcdc283a8edbd1"}
Jul 09 19:16:38.650722 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.650354    2565 scope.go:117] "RemoveContainer" containerID="ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70"
Jul 09 19:16:38.658944 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.658925    2565 scope.go:117] "RemoveContainer" containerID="505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d"
Jul 09 19:16:38.667365 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.667346    2565 scope.go:117] "RemoveContainer" containerID="ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70"
Jul 09 19:16:38.667590 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:16:38.667570    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70\": container with ID starting with ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70 not found: ID does not exist" containerID="ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70"
Jul 09 19:16:38.667637 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.667599    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70"} err="failed to get container status \"ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70\": rpc error: code = NotFound desc = could not find container \"ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70\": container with ID starting with ac1adddb89a72365779af932919a9e80c516ee26cec57c350883c4f0e3893a70 not found: ID does not exist"
Jul 09 19:16:38.667637 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.667615    2565 scope.go:117] "RemoveContainer" containerID="505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d"
Jul 09 19:16:38.667824 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:16:38.667804    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d\": container with ID starting with 505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d not found: ID does not exist" containerID="505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d"
Jul 09 19:16:38.667887 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.667834    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d"} err="failed to get container status \"505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d\": rpc error: code = NotFound desc = could not find container \"505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d\": container with ID starting with 505a0a90b6722689ea7e16a2d6d4334f5219ee1f972ad3b11eeb99aa2ca1662d not found: ID does not exist"
Jul 09 19:16:38.673960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.673936    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"]
Jul 09 19:16:38.678117 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.678095    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-5d5f5494d5-d5r4t"]
Jul 09 19:16:38.868301 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:16:38.868270    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" path="/var/lib/kubelet/pods/7a6cebbf-b9ce-47e2-8cfb-71a03e429dca/volumes"
Jul 09 19:17:03.834122 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.834086    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"]
Jul 09 19:17:03.834472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.834440    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="storage-initializer"
Jul 09 19:17:03.834472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.834452    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="storage-initializer"
Jul 09 19:17:03.834472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.834469    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main"
Jul 09 19:17:03.834584 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.834475    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main"
Jul 09 19:17:03.834584 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.834482    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="436bcaf4-163b-4270-aa0f-8f066d2cf86c" containerName="main"
Jul 09 19:17:03.834584 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.834488    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="436bcaf4-163b-4270-aa0f-8f066d2cf86c" containerName="main"
Jul 09 19:17:03.834584 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.834538    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6cebbf-b9ce-47e2-8cfb-71a03e429dca" containerName="main"
Jul 09 19:17:03.834584 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.834548    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="436bcaf4-163b-4270-aa0f-8f066d2cf86c" containerName="main"
Jul 09 19:17:03.837966 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.837944    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.841197 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.841166    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-sxnqn\""
Jul 09 19:17:03.842014 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.841996    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 09 19:17:03.855879 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.855852    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"]
Jul 09 19:17:03.874153 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.874122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.874289 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.874165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc532e-0937-4148-bf02-53aeef63c68e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.874289 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.874187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.874289 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.874264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.874411 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.874297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.874411 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.874334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkz7\" (UniqueName: \"kubernetes.io/projected/9adc532e-0937-4148-bf02-53aeef63c68e-kube-api-access-dvkz7\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.975136 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.975090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.975338 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.975145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc532e-0937-4148-bf02-53aeef63c68e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.975338 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.975173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.975338 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.975203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.975338 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.975234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.975338 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.975272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkz7\" (UniqueName: \"kubernetes.io/projected/9adc532e-0937-4148-bf02-53aeef63c68e-kube-api-access-dvkz7\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.975627 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.975559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.975627 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.975604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.975821 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.975796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.977472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.977452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.977752 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.977736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc532e-0937-4148-bf02-53aeef63c68e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:03.986494 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:03.986466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkz7\" (UniqueName: \"kubernetes.io/projected/9adc532e-0937-4148-bf02-53aeef63c68e-kube-api-access-dvkz7\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:04.136342 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.136259    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"]
Jul 09 19:17:04.139760 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.139743    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.143067 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.143045    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-pl648\""
Jul 09 19:17:04.147303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.147277    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:04.163458 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.163421    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"]
Jul 09 19:17:04.177029 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.176997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/480f9148-c822-4714-b211-ae3cccc9da27-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.177168 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.177039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4t4\" (UniqueName: \"kubernetes.io/projected/480f9148-c822-4714-b211-ae3cccc9da27-kube-api-access-kv4t4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.177168 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.177074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480f9148-c822-4714-b211-ae3cccc9da27-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.277591 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.277557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/480f9148-c822-4714-b211-ae3cccc9da27-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.277777 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.277602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4t4\" (UniqueName: \"kubernetes.io/projected/480f9148-c822-4714-b211-ae3cccc9da27-kube-api-access-kv4t4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.277777 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.277657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480f9148-c822-4714-b211-ae3cccc9da27-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.277990 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.277967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/480f9148-c822-4714-b211-ae3cccc9da27-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.280133 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.280110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480f9148-c822-4714-b211-ae3cccc9da27-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.293322 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.293299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4t4\" (UniqueName: \"kubernetes.io/projected/480f9148-c822-4714-b211-ae3cccc9da27-kube-api-access-kv4t4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.304035 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.304014    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"]
Jul 09 19:17:04.305682 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:17:04.305659    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adc532e_0937_4148_bf02_53aeef63c68e.slice/crio-07c62c79235c8528c5f1e97167c8083aa029b3c79c8bcf19f676ac0cca57caaa WatchSource:0}: Error finding container 07c62c79235c8528c5f1e97167c8083aa029b3c79c8bcf19f676ac0cca57caaa: Status 404 returned error can't find the container with id 07c62c79235c8528c5f1e97167c8083aa029b3c79c8bcf19f676ac0cca57caaa
Jul 09 19:17:04.307554 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.307536    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:17:04.449873 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.449842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.588699 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.588660    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"]
Jul 09 19:17:04.591428 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:17:04.591397    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480f9148_c822_4714_b211_ae3cccc9da27.slice/crio-abd9fe7cc0bb5a91afd7586e2b7571be4840e0a813c71f99292d1dac3cafbbcd WatchSource:0}: Error finding container abd9fe7cc0bb5a91afd7586e2b7571be4840e0a813c71f99292d1dac3cafbbcd: Status 404 returned error can't find the container with id abd9fe7cc0bb5a91afd7586e2b7571be4840e0a813c71f99292d1dac3cafbbcd
Jul 09 19:17:04.771656 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.771560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc" event={"ID":"9adc532e-0937-4148-bf02-53aeef63c68e","Type":"ContainerStarted","Data":"07c62c79235c8528c5f1e97167c8083aa029b3c79c8bcf19f676ac0cca57caaa"}
Jul 09 19:17:04.773427 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.773397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg" event={"ID":"480f9148-c822-4714-b211-ae3cccc9da27","Type":"ContainerStarted","Data":"5833bbe2804cc5c4ab59f02805a020052ff5b708aa2010c825832f59f154b92e"}
Jul 09 19:17:04.773569 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.773448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg" event={"ID":"480f9148-c822-4714-b211-ae3cccc9da27","Type":"ContainerStarted","Data":"abd9fe7cc0bb5a91afd7586e2b7571be4840e0a813c71f99292d1dac3cafbbcd"}
Jul 09 19:17:04.773569 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.773487    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:04.799026 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:04.798959    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg" podStartSLOduration=0.798936373 podStartE2EDuration="798.936373ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:17:04.797111123 +0000 UTC m=+4344.515373890" watchObservedRunningTime="2026-07-09 19:17:04.798936373 +0000 UTC m=+4344.517199138"
Jul 09 19:17:05.779649 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:05.779605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc" event={"ID":"9adc532e-0937-4148-bf02-53aeef63c68e","Type":"ContainerStarted","Data":"a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49"}
Jul 09 19:17:05.780145 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:05.779692    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:06.785661 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:06.785620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc" event={"ID":"9adc532e-0937-4148-bf02-53aeef63c68e","Type":"ContainerStarted","Data":"5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3"}
Jul 09 19:17:06.808643 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:06.808591    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc" podStartSLOduration=3.011680473 podStartE2EDuration="3.808574591s" totalImagesPullingTime="796.894118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:17:06.806325322 +0000 UTC m=+4346.524588120" watchObservedRunningTime="2026-07-09 19:17:06.808574591 +0000 UTC m=+4346.526837355"
Jul 09 19:17:07.147776 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:07.147681    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:14.147960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:14.147918    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:14.160465 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:14.160442    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:14.840370 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:14.840337    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:15.845783 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:15.845688    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:17:20.874976 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.874945    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"]
Jul 09 19:17:20.878769 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.878746    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:20.881378 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.881359    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 19:17:20.881619 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.881598    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 09 19:17:20.881673 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.881613    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 19:17:20.887521 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.887499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"]
Jul 09 19:17:20.927477 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.927439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdng\" (UniqueName: \"kubernetes.io/projected/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kube-api-access-5bdng\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:20.927655 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.927505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-home\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:20.927655 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.927526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:20.927655 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.927565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-model-cache\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:20.927655 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.927597    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:20.927655 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.927638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:20.927973 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.927663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:20.927973 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:20.927739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-dshm\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.028377 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-dshm\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.028580 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdng\" (UniqueName: \"kubernetes.io/projected/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kube-api-access-5bdng\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.028580 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-home\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.028580 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.028580 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-model-cache\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.028580 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.028873 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.028873 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.029036 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028893    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.029036 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.028944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-home\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.029036 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.029014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-model-cache\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.029163 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.029041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.029398 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.029380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.030841 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.030808    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-dshm\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.031043 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.031025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.037019 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.036994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdng\" (UniqueName: \"kubernetes.io/projected/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kube-api-access-5bdng\") pod \"lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.191863 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.191767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:21.336059 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.336032    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"]
Jul 09 19:17:21.337621 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:17:21.337591    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e9ad88_622b_4b31_b6a5_c26845b2d7a5.slice/crio-69b1a2b906fd41fbf29f5467991c4d48ca67e505cd5bd8e80b5545a9b147f7f3 WatchSource:0}: Error finding container 69b1a2b906fd41fbf29f5467991c4d48ca67e505cd5bd8e80b5545a9b147f7f3: Status 404 returned error can't find the container with id 69b1a2b906fd41fbf29f5467991c4d48ca67e505cd5bd8e80b5545a9b147f7f3
Jul 09 19:17:21.862855 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.862823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" event={"ID":"87e9ad88-622b-4b31-b6a5-c26845b2d7a5","Type":"ContainerStarted","Data":"ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f"}
Jul 09 19:17:21.863068 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:21.862875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" event={"ID":"87e9ad88-622b-4b31-b6a5-c26845b2d7a5","Type":"ContainerStarted","Data":"69b1a2b906fd41fbf29f5467991c4d48ca67e505cd5bd8e80b5545a9b147f7f3"}
Jul 09 19:17:25.887209 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:25.887175    2565 generic.go:356] "Generic (PLEG): container finished" podID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerID="ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f" exitCode=0
Jul 09 19:17:25.887593 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:25.887242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" event={"ID":"87e9ad88-622b-4b31-b6a5-c26845b2d7a5","Type":"ContainerDied","Data":"ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f"}
Jul 09 19:17:26.893316 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:26.893279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" event={"ID":"87e9ad88-622b-4b31-b6a5-c26845b2d7a5","Type":"ContainerStarted","Data":"d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461"}
Jul 09 19:17:26.930534 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:26.930482    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" podStartSLOduration=3.301749514 podStartE2EDuration="6.930464565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.628715051s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:17:26.92992638 +0000 UTC m=+4366.648189144" watchObservedRunningTime="2026-07-09 19:17:26.930464565 +0000 UTC m=+4366.648727328"
Jul 09 19:17:27.192553 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:27.192455    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:31.192173 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:31.192129    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:17:31.193822 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:31.193785    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 09 19:17:35.781851 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:35.781813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:17:41.192954 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:41.192897    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 09 19:17:51.192318 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:17:51.192276    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 09 19:18:01.193092 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:01.193046    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 09 19:18:11.193270 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:11.193218    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 09 19:18:21.193093 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:21.193040    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 09 19:18:31.192662 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:31.192612    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 09 19:18:41.118080 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.118045    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"]
Jul 09 19:18:41.124356 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.124328    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"]
Jul 09 19:18:41.128343 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:18:41.128300    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:18:41.128343 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:18:41.128343 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:18:41.128343 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="5833bbe2804cc5c4ab59f02805a020052ff5b708aa2010c825832f59f154b92e" cmd=["/bin/sleep","15"]
Jul 09 19:18:41.128588 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.128371    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg" message="" err=<
Jul 09 19:18:41.128588 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:18:41.128588 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:18:41.128588 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:18:41.128588 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:18:41.128392    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:18:41.128588 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:18:41.128588 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:18:41.128588 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg" podUID="480f9148-c822-4714-b211-ae3cccc9da27" containerName="main" containerID="cri-o://5833bbe2804cc5c4ab59f02805a020052ff5b708aa2010c825832f59f154b92e"
Jul 09 19:18:41.128588 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.128435    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg" podUID="480f9148-c822-4714-b211-ae3cccc9da27" containerName="main" containerID="cri-o://5833bbe2804cc5c4ab59f02805a020052ff5b708aa2010c825832f59f154b92e" gracePeriod=60
Jul 09 19:18:41.202826 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.202796    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:18:41.210537 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.210517    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:18:41.261671 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.261639    2565 generic.go:356] "Generic (PLEG): container finished" podID="480f9148-c822-4714-b211-ae3cccc9da27" containerID="5833bbe2804cc5c4ab59f02805a020052ff5b708aa2010c825832f59f154b92e" exitCode=0
Jul 09 19:18:41.261860 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.261710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg" event={"ID":"480f9148-c822-4714-b211-ae3cccc9da27","Type":"ContainerDied","Data":"5833bbe2804cc5c4ab59f02805a020052ff5b708aa2010c825832f59f154b92e"}
Jul 09 19:18:41.376508 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.376445    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:18:41.490085 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.490053    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/480f9148-c822-4714-b211-ae3cccc9da27-tokenizer-uds\") pod \"480f9148-c822-4714-b211-ae3cccc9da27\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") "
Jul 09 19:18:41.490256 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.490096    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv4t4\" (UniqueName: \"kubernetes.io/projected/480f9148-c822-4714-b211-ae3cccc9da27-kube-api-access-kv4t4\") pod \"480f9148-c822-4714-b211-ae3cccc9da27\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") "
Jul 09 19:18:41.490256 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.490167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480f9148-c822-4714-b211-ae3cccc9da27-tls-certs\") pod \"480f9148-c822-4714-b211-ae3cccc9da27\" (UID: \"480f9148-c822-4714-b211-ae3cccc9da27\") "
Jul 09 19:18:41.490339 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.490308    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480f9148-c822-4714-b211-ae3cccc9da27-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "480f9148-c822-4714-b211-ae3cccc9da27" (UID: "480f9148-c822-4714-b211-ae3cccc9da27"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:18:41.490445 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.490430    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/480f9148-c822-4714-b211-ae3cccc9da27-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:18:41.492260 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.492225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480f9148-c822-4714-b211-ae3cccc9da27-kube-api-access-kv4t4" (OuterVolumeSpecName: "kube-api-access-kv4t4") pod "480f9148-c822-4714-b211-ae3cccc9da27" (UID: "480f9148-c822-4714-b211-ae3cccc9da27"). InnerVolumeSpecName "kube-api-access-kv4t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:18:41.492359 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.492283    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480f9148-c822-4714-b211-ae3cccc9da27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "480f9148-c822-4714-b211-ae3cccc9da27" (UID: "480f9148-c822-4714-b211-ae3cccc9da27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:18:41.591656 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.591625    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480f9148-c822-4714-b211-ae3cccc9da27-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:18:41.591656 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:41.591649    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4t4\" (UniqueName: \"kubernetes.io/projected/480f9148-c822-4714-b211-ae3cccc9da27-kube-api-access-kv4t4\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:18:42.267678 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:42.267651    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"
Jul 09 19:18:42.268215 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:42.267685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg" event={"ID":"480f9148-c822-4714-b211-ae3cccc9da27","Type":"ContainerDied","Data":"abd9fe7cc0bb5a91afd7586e2b7571be4840e0a813c71f99292d1dac3cafbbcd"}
Jul 09 19:18:42.268215 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:42.267743    2565 scope.go:117] "RemoveContainer" containerID="5833bbe2804cc5c4ab59f02805a020052ff5b708aa2010c825832f59f154b92e"
Jul 09 19:18:42.289849 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:42.289819    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"]
Jul 09 19:18:42.292112 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:42.292089    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59sfggg"]
Jul 09 19:18:42.868632 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:42.868591    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480f9148-c822-4714-b211-ae3cccc9da27" path="/var/lib/kubelet/pods/480f9148-c822-4714-b211-ae3cccc9da27/volumes"
Jul 09 19:18:56.147786 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.147720    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc" podUID="9adc532e-0937-4148-bf02-53aeef63c68e" containerName="main" containerID="cri-o://5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3" gracePeriod=45
Jul 09 19:18:56.263595 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.263558    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc" podUID="9adc532e-0937-4148-bf02-53aeef63c68e" containerName="llm-d-routing-sidecar" containerID="cri-o://a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49" gracePeriod=45
Jul 09 19:18:56.342184 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.342149    2565 generic.go:356] "Generic (PLEG): container finished" podID="9adc532e-0937-4148-bf02-53aeef63c68e" containerID="5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3" exitCode=0
Jul 09 19:18:56.342355 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.342227    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc" event={"ID":"9adc532e-0937-4148-bf02-53aeef63c68e","Type":"ContainerDied","Data":"5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3"}
Jul 09 19:18:56.516145 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.516120    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:18:56.631072 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631029    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc532e-0937-4148-bf02-53aeef63c68e-tls-certs\") pod \"9adc532e-0937-4148-bf02-53aeef63c68e\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") "
Jul 09 19:18:56.631266 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631097    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-model-cache\") pod \"9adc532e-0937-4148-bf02-53aeef63c68e\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") "
Jul 09 19:18:56.631266 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631140    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-tmp-dir\") pod \"9adc532e-0937-4148-bf02-53aeef63c68e\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") "
Jul 09 19:18:56.631266 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvkz7\" (UniqueName: \"kubernetes.io/projected/9adc532e-0937-4148-bf02-53aeef63c68e-kube-api-access-dvkz7\") pod \"9adc532e-0937-4148-bf02-53aeef63c68e\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") "
Jul 09 19:18:56.631266 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631262    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-home\") pod \"9adc532e-0937-4148-bf02-53aeef63c68e\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") "
Jul 09 19:18:56.631505 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631335    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-dshm\") pod \"9adc532e-0937-4148-bf02-53aeef63c68e\" (UID: \"9adc532e-0937-4148-bf02-53aeef63c68e\") "
Jul 09 19:18:56.631505 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631359    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9adc532e-0937-4148-bf02-53aeef63c68e" (UID: "9adc532e-0937-4148-bf02-53aeef63c68e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:18:56.631505 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9adc532e-0937-4148-bf02-53aeef63c68e" (UID: "9adc532e-0937-4148-bf02-53aeef63c68e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:18:56.631661 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631543    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-home" (OuterVolumeSpecName: "home") pod "9adc532e-0937-4148-bf02-53aeef63c68e" (UID: "9adc532e-0937-4148-bf02-53aeef63c68e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:18:56.631661 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631628    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:18:56.631661 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631647    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:18:56.631661 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.631658    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:18:56.633432 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.633404    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-dshm" (OuterVolumeSpecName: "dshm") pod "9adc532e-0937-4148-bf02-53aeef63c68e" (UID: "9adc532e-0937-4148-bf02-53aeef63c68e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:18:56.633822 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.633794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adc532e-0937-4148-bf02-53aeef63c68e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9adc532e-0937-4148-bf02-53aeef63c68e" (UID: "9adc532e-0937-4148-bf02-53aeef63c68e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:18:56.633822 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.633796    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adc532e-0937-4148-bf02-53aeef63c68e-kube-api-access-dvkz7" (OuterVolumeSpecName: "kube-api-access-dvkz7") pod "9adc532e-0937-4148-bf02-53aeef63c68e" (UID: "9adc532e-0937-4148-bf02-53aeef63c68e"). InnerVolumeSpecName "kube-api-access-dvkz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:18:56.732883 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.732853    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkz7\" (UniqueName: \"kubernetes.io/projected/9adc532e-0937-4148-bf02-53aeef63c68e-kube-api-access-dvkz7\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:18:56.732883 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.732882    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adc532e-0937-4148-bf02-53aeef63c68e-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:18:56.732883 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:56.732892    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adc532e-0937-4148-bf02-53aeef63c68e-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:18:57.347797 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.347768    2565 generic.go:356] "Generic (PLEG): container finished" podID="9adc532e-0937-4148-bf02-53aeef63c68e" containerID="a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49" exitCode=0
Jul 09 19:18:57.348249 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.347832    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"
Jul 09 19:18:57.348249 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.347843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc" event={"ID":"9adc532e-0937-4148-bf02-53aeef63c68e","Type":"ContainerDied","Data":"a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49"}
Jul 09 19:18:57.348249 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.347892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc" event={"ID":"9adc532e-0937-4148-bf02-53aeef63c68e","Type":"ContainerDied","Data":"07c62c79235c8528c5f1e97167c8083aa029b3c79c8bcf19f676ac0cca57caaa"}
Jul 09 19:18:57.348249 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.347946    2565 scope.go:117] "RemoveContainer" containerID="5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3"
Jul 09 19:18:57.356962 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.356944    2565 scope.go:117] "RemoveContainer" containerID="a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49"
Jul 09 19:18:57.365835 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.365587    2565 scope.go:117] "RemoveContainer" containerID="5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3"
Jul 09 19:18:57.365978 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:18:57.365952    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3\": container with ID starting with 5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3 not found: ID does not exist" containerID="5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3"
Jul 09 19:18:57.366039 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.365998    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3"} err="failed to get container status \"5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3\": rpc error: code = NotFound desc = could not find container \"5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3\": container with ID starting with 5d84dfa3391e1493aaac3f6934acd381b7005bbcb0f840790184221d9dc345e3 not found: ID does not exist"
Jul 09 19:18:57.366039 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.366020    2565 scope.go:117] "RemoveContainer" containerID="a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49"
Jul 09 19:18:57.366329 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:18:57.366305    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49\": container with ID starting with a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49 not found: ID does not exist" containerID="a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49"
Jul 09 19:18:57.366410 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.366337    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49"} err="failed to get container status \"a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49\": rpc error: code = NotFound desc = could not find container \"a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49\": container with ID starting with a9cb3909acadb7379cf1690ca10f06d5b23b2140f94e93be7908fc88fc1d4c49 not found: ID does not exist"
Jul 09 19:18:57.367609 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.367589    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"]
Jul 09 19:18:57.372863 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:57.372843    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-pw7lc"]
Jul 09 19:18:58.867943 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:18:58.867880    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adc532e-0937-4148-bf02-53aeef63c68e" path="/var/lib/kubelet/pods/9adc532e-0937-4148-bf02-53aeef63c68e/volumes"
Jul 09 19:19:33.504231 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504155    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"]
Jul 09 19:19:33.504685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504607    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adc532e-0937-4148-bf02-53aeef63c68e" containerName="llm-d-routing-sidecar"
Jul 09 19:19:33.504685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504626    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adc532e-0937-4148-bf02-53aeef63c68e" containerName="llm-d-routing-sidecar"
Jul 09 19:19:33.504685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504646    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480f9148-c822-4714-b211-ae3cccc9da27" containerName="main"
Jul 09 19:19:33.504685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504655    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="480f9148-c822-4714-b211-ae3cccc9da27" containerName="main"
Jul 09 19:19:33.504685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504680    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adc532e-0937-4148-bf02-53aeef63c68e" containerName="main"
Jul 09 19:19:33.504685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504691    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adc532e-0937-4148-bf02-53aeef63c68e" containerName="main"
Jul 09 19:19:33.504929 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504784    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="480f9148-c822-4714-b211-ae3cccc9da27" containerName="main"
Jul 09 19:19:33.504929 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504799    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adc532e-0937-4148-bf02-53aeef63c68e" containerName="llm-d-routing-sidecar"
Jul 09 19:19:33.504929 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.504809    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adc532e-0937-4148-bf02-53aeef63c68e" containerName="main"
Jul 09 19:19:33.507825 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.507804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.510679 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.510651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 09 19:19:33.519811 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.519785    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"]
Jul 09 19:19:33.685365 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.685329    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484e7a29-5998-440c-a9e9-8f0c50238a73-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.685365 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.685372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.685622 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.685416    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.685622 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.685511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbng\" (UniqueName: \"kubernetes.io/projected/484e7a29-5998-440c-a9e9-8f0c50238a73-kube-api-access-dlbng\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.685622 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.685558    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.685622 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.685581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.786748 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.786653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.786748 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.786716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbng\" (UniqueName: \"kubernetes.io/projected/484e7a29-5998-440c-a9e9-8f0c50238a73-kube-api-access-dlbng\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.786748 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.786751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.787042 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.786768    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.787042 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.786804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484e7a29-5998-440c-a9e9-8f0c50238a73-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.787042 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.786946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.787211 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.787168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.787270 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.787222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.787270 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.787244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.789086 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.789068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.789460 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.789441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484e7a29-5998-440c-a9e9-8f0c50238a73-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.796599 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.796567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbng\" (UniqueName: \"kubernetes.io/projected/484e7a29-5998-440c-a9e9-8f0c50238a73-kube-api-access-dlbng\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.818407 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.818379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:33.927054 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.927023    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"]
Jul 09 19:19:33.930693 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.930668    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:33.940551 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.940522    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-mgb4p\""
Jul 09 19:19:33.952627 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.951089    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"]
Jul 09 19:19:33.982177 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:33.982151    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"]
Jul 09 19:19:33.984698 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:19:33.984665    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484e7a29_5998_440c_a9e9_8f0c50238a73.slice/crio-2430fd6134a708ffbbe45818862391110b18a282fb95f16fbdca404d656815b9 WatchSource:0}: Error finding container 2430fd6134a708ffbbe45818862391110b18a282fb95f16fbdca404d656815b9: Status 404 returned error can't find the container with id 2430fd6134a708ffbbe45818862391110b18a282fb95f16fbdca404d656815b9
Jul 09 19:19:34.089340 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.089298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af354ac7-d71a-4744-8a24-cfefce87dccb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.089526 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.089360    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/af354ac7-d71a-4744-8a24-cfefce87dccb-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.089526 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.089428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vq8j\" (UniqueName: \"kubernetes.io/projected/af354ac7-d71a-4744-8a24-cfefce87dccb-kube-api-access-5vq8j\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.190373 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.190325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af354ac7-d71a-4744-8a24-cfefce87dccb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.190584 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.190416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/af354ac7-d71a-4744-8a24-cfefce87dccb-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.190584 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.190449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vq8j\" (UniqueName: \"kubernetes.io/projected/af354ac7-d71a-4744-8a24-cfefce87dccb-kube-api-access-5vq8j\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.190855 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.190830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/af354ac7-d71a-4744-8a24-cfefce87dccb-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.192843 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.192819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af354ac7-d71a-4744-8a24-cfefce87dccb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.200737 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.200714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vq8j\" (UniqueName: \"kubernetes.io/projected/af354ac7-d71a-4744-8a24-cfefce87dccb-kube-api-access-5vq8j\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.242468 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.242424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.381231 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.381197    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"]
Jul 09 19:19:34.383729 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:19:34.383700    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf354ac7_d71a_4744_8a24_cfefce87dccb.slice/crio-4b190ba962e69f5eda11555d691328a60a139feb349eeea64c4ee3211d44cf72 WatchSource:0}: Error finding container 4b190ba962e69f5eda11555d691328a60a139feb349eeea64c4ee3211d44cf72: Status 404 returned error can't find the container with id 4b190ba962e69f5eda11555d691328a60a139feb349eeea64c4ee3211d44cf72
Jul 09 19:19:34.562160 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.562053    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2" event={"ID":"484e7a29-5998-440c-a9e9-8f0c50238a73","Type":"ContainerStarted","Data":"c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e"}
Jul 09 19:19:34.562160 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.562126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2" event={"ID":"484e7a29-5998-440c-a9e9-8f0c50238a73","Type":"ContainerStarted","Data":"2430fd6134a708ffbbe45818862391110b18a282fb95f16fbdca404d656815b9"}
Jul 09 19:19:34.563565 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.563537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf" event={"ID":"af354ac7-d71a-4744-8a24-cfefce87dccb","Type":"ContainerStarted","Data":"018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de"}
Jul 09 19:19:34.563686 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.563585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf" event={"ID":"af354ac7-d71a-4744-8a24-cfefce87dccb","Type":"ContainerStarted","Data":"4b190ba962e69f5eda11555d691328a60a139feb349eeea64c4ee3211d44cf72"}
Jul 09 19:19:34.563686 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.563608    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:19:34.583805 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.583753    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2" podStartSLOduration=1.583737379 podStartE2EDuration="1.583737379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:19:34.581645754 +0000 UTC m=+4494.299908519" watchObservedRunningTime="2026-07-09 19:19:34.583737379 +0000 UTC m=+4494.302000143"
Jul 09 19:19:34.599562 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.599507    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf" podStartSLOduration=1.599478403 podStartE2EDuration="1.599478403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:19:34.598089331 +0000 UTC m=+4494.316352107" watchObservedRunningTime="2026-07-09 19:19:34.599478403 +0000 UTC m=+4494.317741167"
Jul 09 19:19:34.819217 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:34.819138    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:41.068634 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:41.068524    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:19:41.087060 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:41.078261    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:19:43.818978 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:43.818927    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:43.831514 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:43.831485    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:44.629159 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:44.629127    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:19:58.279253 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:19:58.279219    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"]
Jul 09 19:20:05.571222 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:05.571192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:20:13.302941 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:13.302854    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main" containerID="cri-o://d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461" gracePeriod=45
Jul 09 19:20:13.960430 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:13.960403    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:20:14.054118 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054032    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-dshm\") pod \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") "
Jul 09 19:20:14.054118 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054070    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bdng\" (UniqueName: \"kubernetes.io/projected/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kube-api-access-5bdng\") pod \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") "
Jul 09 19:20:14.054118 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tls-certs\") pod \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") "
Jul 09 19:20:14.054396 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054126    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kserve-provision-location\") pod \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") "
Jul 09 19:20:14.054396 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054158    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tmp-dir\") pod \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") "
Jul 09 19:20:14.054396 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054199    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-model-cache\") pod \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") "
Jul 09 19:20:14.054396 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054246    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-cabundle-cert\") pod \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") "
Jul 09 19:20:14.054396 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054291    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-home\") pod \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\" (UID: \"87e9ad88-622b-4b31-b6a5-c26845b2d7a5\") "
Jul 09 19:20:14.054658 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87e9ad88-622b-4b31-b6a5-c26845b2d7a5" (UID: "87e9ad88-622b-4b31-b6a5-c26845b2d7a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:20:14.054883 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054856    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "87e9ad88-622b-4b31-b6a5-c26845b2d7a5" (UID: "87e9ad88-622b-4b31-b6a5-c26845b2d7a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:20:14.055091 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.054979    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-home" (OuterVolumeSpecName: "home") pod "87e9ad88-622b-4b31-b6a5-c26845b2d7a5" (UID: "87e9ad88-622b-4b31-b6a5-c26845b2d7a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:20:14.056400 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.056374    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-dshm" (OuterVolumeSpecName: "dshm") pod "87e9ad88-622b-4b31-b6a5-c26845b2d7a5" (UID: "87e9ad88-622b-4b31-b6a5-c26845b2d7a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:20:14.056491 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.056444    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kube-api-access-5bdng" (OuterVolumeSpecName: "kube-api-access-5bdng") pod "87e9ad88-622b-4b31-b6a5-c26845b2d7a5" (UID: "87e9ad88-622b-4b31-b6a5-c26845b2d7a5"). InnerVolumeSpecName "kube-api-access-5bdng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:20:14.056654 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.056633    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87e9ad88-622b-4b31-b6a5-c26845b2d7a5" (UID: "87e9ad88-622b-4b31-b6a5-c26845b2d7a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:20:14.065471 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.065446    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87e9ad88-622b-4b31-b6a5-c26845b2d7a5" (UID: "87e9ad88-622b-4b31-b6a5-c26845b2d7a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:20:14.120229 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.120188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87e9ad88-622b-4b31-b6a5-c26845b2d7a5" (UID: "87e9ad88-622b-4b31-b6a5-c26845b2d7a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:20:14.155627 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.155593    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:20:14.155627 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.155622    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bdng\" (UniqueName: \"kubernetes.io/projected/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kube-api-access-5bdng\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:20:14.155627 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.155633    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:20:14.155855 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.155645    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:20:14.155855 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.155675    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:20:14.155855 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.155684    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:20:14.155855 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.155692    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-cabundle-cert\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:20:14.155855 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.155700    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e9ad88-622b-4b31-b6a5-c26845b2d7a5-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:20:14.759549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.759519    2565 generic.go:356] "Generic (PLEG): container finished" podID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerID="d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461" exitCode=0
Jul 09 19:20:14.759956 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.759571    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"
Jul 09 19:20:14.759956 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.759604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" event={"ID":"87e9ad88-622b-4b31-b6a5-c26845b2d7a5","Type":"ContainerDied","Data":"d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461"}
Jul 09 19:20:14.759956 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.759665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs" event={"ID":"87e9ad88-622b-4b31-b6a5-c26845b2d7a5","Type":"ContainerDied","Data":"69b1a2b906fd41fbf29f5467991c4d48ca67e505cd5bd8e80b5545a9b147f7f3"}
Jul 09 19:20:14.759956 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.759697    2565 scope.go:117] "RemoveContainer" containerID="d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461"
Jul 09 19:20:14.768602 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.768584    2565 scope.go:117] "RemoveContainer" containerID="ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f"
Jul 09 19:20:14.776280 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.776263    2565 scope.go:117] "RemoveContainer" containerID="d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461"
Jul 09 19:20:14.776512 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:20:14.776493    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461\": container with ID starting with d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461 not found: ID does not exist" containerID="d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461"
Jul 09 19:20:14.776558 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.776520    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461"} err="failed to get container status \"d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461\": rpc error: code = NotFound desc = could not find container \"d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461\": container with ID starting with d41caa21214c9d35060b446545f2c6b3124d4deb962ed36f01a804f33a063461 not found: ID does not exist"
Jul 09 19:20:14.776558 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.776538    2565 scope.go:117] "RemoveContainer" containerID="ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f"
Jul 09 19:20:14.776781 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:20:14.776763    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f\": container with ID starting with ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f not found: ID does not exist" containerID="ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f"
Jul 09 19:20:14.776837 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.776790    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f"} err="failed to get container status \"ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f\": rpc error: code = NotFound desc = could not find container \"ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f\": container with ID starting with ba5166e36408788e1930ed4801b1ff00318e7338a73badd0d6dc80c53b36288f not found: ID does not exist"
Jul 09 19:20:14.790818 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.790792    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"]
Jul 09 19:20:14.807641 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.807615    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-67c749d4b6-zhgzs"]
Jul 09 19:20:14.869067 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:14.869037    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" path="/var/lib/kubelet/pods/87e9ad88-622b-4b31-b6a5-c26845b2d7a5/volumes"
Jul 09 19:20:58.494236 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.494205    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"]
Jul 09 19:20:58.494685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.494566    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="storage-initializer"
Jul 09 19:20:58.494685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.494580    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="storage-initializer"
Jul 09 19:20:58.494685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.494614    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main"
Jul 09 19:20:58.494685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.494621    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main"
Jul 09 19:20:58.494685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.494683    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e9ad88-622b-4b31-b6a5-c26845b2d7a5" containerName="main"
Jul 09 19:20:58.498954 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.498932    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.501989 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.501972    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 19:20:58.502063 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.502043    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 09 19:20:58.509323 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.509298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"]
Jul 09 19:20:58.624290 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.624252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da907bc-769e-461c-9b25-e6f09c204add-tls-certs\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.624466 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.624297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mnt\" (UniqueName: \"kubernetes.io/projected/7da907bc-769e-461c-9b25-e6f09c204add-kube-api-access-n7mnt\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.624466 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.624337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-dshm\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.624466 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.624359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-kserve-provision-location\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.624466 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.624426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-model-cache\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.624669 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.624468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-tmp-dir\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.624669 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.624512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-home\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.725425 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-model-cache\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.725625 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-tmp-dir\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.725625 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-home\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.725625 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da907bc-769e-461c-9b25-e6f09c204add-tls-certs\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.725625 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mnt\" (UniqueName: \"kubernetes.io/projected/7da907bc-769e-461c-9b25-e6f09c204add-kube-api-access-n7mnt\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.725625 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-dshm\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.725931 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-kserve-provision-location\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.725931 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725828    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-model-cache\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.725931 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725859    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-tmp-dir\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.726102 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.725937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-home\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.726153 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.726101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-kserve-provision-location\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.727982 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.727952    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-dshm\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.728129 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.728108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da907bc-769e-461c-9b25-e6f09c204add-tls-certs\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.736761 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.736736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mnt\" (UniqueName: \"kubernetes.io/projected/7da907bc-769e-461c-9b25-e6f09c204add-kube-api-access-n7mnt\") pod \"tls-verification-test-kserve-59fd59cf4d-r5fjc\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.782963 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.782858    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"]
Jul 09 19:20:58.786596 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.786576    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.789350 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.789323    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-nmcfn\""
Jul 09 19:20:58.797492 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.797469    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"]
Jul 09 19:20:58.809445 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.809418    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:20:58.826227 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.826202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.826368 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.826246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.826368 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.826307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7tq\" (UniqueName: \"kubernetes.io/projected/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kube-api-access-mb7tq\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.826487 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.826419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.927674 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.927640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.927862 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.927699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.927862 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.927735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.927862 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.927773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7tq\" (UniqueName: \"kubernetes.io/projected/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kube-api-access-mb7tq\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.928247 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.928224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.928310 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.928248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.930241 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.930221    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.941012 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.940987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7tq\" (UniqueName: \"kubernetes.io/projected/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kube-api-access-mb7tq\") pod \"tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:58.957183 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.957160    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"]
Jul 09 19:20:58.959100 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:20:58.959068    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da907bc_769e_461c_9b25_e6f09c204add.slice/crio-04654ed49820065a471b03dae0b152af242834c468a5378fe6e508cda8e1455b WatchSource:0}: Error finding container 04654ed49820065a471b03dae0b152af242834c468a5378fe6e508cda8e1455b: Status 404 returned error can't find the container with id 04654ed49820065a471b03dae0b152af242834c468a5378fe6e508cda8e1455b
Jul 09 19:20:58.966395 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:58.966368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" event={"ID":"7da907bc-769e-461c-9b25-e6f09c204add","Type":"ContainerStarted","Data":"04654ed49820065a471b03dae0b152af242834c468a5378fe6e508cda8e1455b"}
Jul 09 19:20:59.097479 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:59.097392    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:20:59.242130 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:59.241683    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"]
Jul 09 19:20:59.243543 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:20:59.243497    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcb4019_0f5f_48f4_8624_b5a645c3bec2.slice/crio-161a42fae890f52bd12bda492dcbacb6396bb857c6b4fbf2cc3f1950771d9775 WatchSource:0}: Error finding container 161a42fae890f52bd12bda492dcbacb6396bb857c6b4fbf2cc3f1950771d9775: Status 404 returned error can't find the container with id 161a42fae890f52bd12bda492dcbacb6396bb857c6b4fbf2cc3f1950771d9775
Jul 09 19:20:59.980621 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:59.980582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" event={"ID":"7da907bc-769e-461c-9b25-e6f09c204add","Type":"ContainerStarted","Data":"9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43"}
Jul 09 19:20:59.983475 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:59.983434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" event={"ID":"adcb4019-0f5f-48f4-8624-b5a645c3bec2","Type":"ContainerStarted","Data":"3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636"}
Jul 09 19:20:59.983694 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:20:59.983505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" event={"ID":"adcb4019-0f5f-48f4-8624-b5a645c3bec2","Type":"ContainerStarted","Data":"161a42fae890f52bd12bda492dcbacb6396bb857c6b4fbf2cc3f1950771d9775"}
Jul 09 19:21:00.994652 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:00.994610    2565 generic.go:356] "Generic (PLEG): container finished" podID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" containerID="3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636" exitCode=0
Jul 09 19:21:00.995216 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:00.994681    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" event={"ID":"adcb4019-0f5f-48f4-8624-b5a645c3bec2","Type":"ContainerDied","Data":"3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636"}
Jul 09 19:21:02.002011 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:02.001968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" event={"ID":"adcb4019-0f5f-48f4-8624-b5a645c3bec2","Type":"ContainerStarted","Data":"61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17"}
Jul 09 19:21:02.002498 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:02.002067    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:21:02.023614 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:02.023565    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" podStartSLOduration=3.335773883 podStartE2EDuration="4.023550306s" totalImagesPullingTime="0s" totalInitContainerRuntime="687.776423ms" isStatefulPod=false podCreationTimestamp="2026-07-09 19:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:21:02.022080066 +0000 UTC m=+4581.740342832" watchObservedRunningTime="2026-07-09 19:21:02.023550306 +0000 UTC m=+4581.741813069"
Jul 09 19:21:03.007494 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.007454    2565 generic.go:356] "Generic (PLEG): container finished" podID="7da907bc-769e-461c-9b25-e6f09c204add" containerID="9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43" exitCode=0
Jul 09 19:21:03.007868 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.007529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" event={"ID":"7da907bc-769e-461c-9b25-e6f09c204add","Type":"ContainerDied","Data":"9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43"}
Jul 09 19:21:03.314388 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.314299    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"]
Jul 09 19:21:03.321176 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.321145    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"]
Jul 09 19:21:03.329630 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:21:03.329589    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:21:03.329630 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:21:03.329630 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:21:03.329630 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de" cmd=["/bin/sleep","15"]
Jul 09 19:21:03.329982 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.329644    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf" message="" err=<
Jul 09 19:21:03.329982 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:21:03.329982 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:21:03.329982 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:21:03.329982 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:21:03.329656    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:21:03.329982 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:21:03.329982 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:21:03.329982 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf" podUID="af354ac7-d71a-4744-8a24-cfefce87dccb" containerName="main" containerID="cri-o://018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de"
Jul 09 19:21:03.329982 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.329690    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf" podUID="af354ac7-d71a-4744-8a24-cfefce87dccb" containerName="main" containerID="cri-o://018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de" gracePeriod=60
Jul 09 19:21:03.593726 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.593690    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:21:03.675603 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.675567    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vq8j\" (UniqueName: \"kubernetes.io/projected/af354ac7-d71a-4744-8a24-cfefce87dccb-kube-api-access-5vq8j\") pod \"af354ac7-d71a-4744-8a24-cfefce87dccb\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") "
Jul 09 19:21:03.675778 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.675671    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af354ac7-d71a-4744-8a24-cfefce87dccb-tls-certs\") pod \"af354ac7-d71a-4744-8a24-cfefce87dccb\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") "
Jul 09 19:21:03.675778 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.675721    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/af354ac7-d71a-4744-8a24-cfefce87dccb-tokenizer-uds\") pod \"af354ac7-d71a-4744-8a24-cfefce87dccb\" (UID: \"af354ac7-d71a-4744-8a24-cfefce87dccb\") "
Jul 09 19:21:03.676059 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.676035    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af354ac7-d71a-4744-8a24-cfefce87dccb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "af354ac7-d71a-4744-8a24-cfefce87dccb" (UID: "af354ac7-d71a-4744-8a24-cfefce87dccb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:21:03.677668 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.677640    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af354ac7-d71a-4744-8a24-cfefce87dccb-kube-api-access-5vq8j" (OuterVolumeSpecName: "kube-api-access-5vq8j") pod "af354ac7-d71a-4744-8a24-cfefce87dccb" (UID: "af354ac7-d71a-4744-8a24-cfefce87dccb"). InnerVolumeSpecName "kube-api-access-5vq8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:21:03.677771 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.677729    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af354ac7-d71a-4744-8a24-cfefce87dccb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af354ac7-d71a-4744-8a24-cfefce87dccb" (UID: "af354ac7-d71a-4744-8a24-cfefce87dccb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:21:03.777288 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.777248    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af354ac7-d71a-4744-8a24-cfefce87dccb-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:21:03.777540 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.777523    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/af354ac7-d71a-4744-8a24-cfefce87dccb-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:21:03.777644 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:03.777630    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vq8j\" (UniqueName: \"kubernetes.io/projected/af354ac7-d71a-4744-8a24-cfefce87dccb-kube-api-access-5vq8j\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:21:04.013765 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.013723    2565 generic.go:356] "Generic (PLEG): container finished" podID="af354ac7-d71a-4744-8a24-cfefce87dccb" containerID="018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de" exitCode=0
Jul 09 19:21:04.013765 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.013768    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"
Jul 09 19:21:04.014458 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.013821    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf" event={"ID":"af354ac7-d71a-4744-8a24-cfefce87dccb","Type":"ContainerDied","Data":"018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de"}
Jul 09 19:21:04.014458 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.013884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf" event={"ID":"af354ac7-d71a-4744-8a24-cfefce87dccb","Type":"ContainerDied","Data":"4b190ba962e69f5eda11555d691328a60a139feb349eeea64c4ee3211d44cf72"}
Jul 09 19:21:04.014458 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.013933    2565 scope.go:117] "RemoveContainer" containerID="018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de"
Jul 09 19:21:04.016078 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.016054    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" event={"ID":"7da907bc-769e-461c-9b25-e6f09c204add","Type":"ContainerStarted","Data":"ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2"}
Jul 09 19:21:04.023797 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.023776    2565 scope.go:117] "RemoveContainer" containerID="018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de"
Jul 09 19:21:04.024125 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:21:04.024104    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de\": container with ID starting with 018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de not found: ID does not exist" containerID="018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de"
Jul 09 19:21:04.024205 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.024138    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de"} err="failed to get container status \"018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de\": rpc error: code = NotFound desc = could not find container \"018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de\": container with ID starting with 018ee15dd9477740d65edcabc8a3058ef8a922becaed12fe08efc083c64645de not found: ID does not exist"
Jul 09 19:21:04.045024 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.044947    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podStartSLOduration=2.755694877 podStartE2EDuration="6.044928402s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.289233525s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:21:04.043737284 +0000 UTC m=+4583.762000059" watchObservedRunningTime="2026-07-09 19:21:04.044928402 +0000 UTC m=+4583.763191166"
Jul 09 19:21:04.066098 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.066058    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"]
Jul 09 19:21:04.072314 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.072285    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedt7rrf"]
Jul 09 19:21:04.810115 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.810080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:21:04.869493 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:04.869461    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af354ac7-d71a-4744-8a24-cfefce87dccb" path="/var/lib/kubelet/pods/af354ac7-d71a-4744-8a24-cfefce87dccb/volumes"
Jul 09 19:21:08.809673 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:08.809634    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:21:08.811402 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:08.811372    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:21:18.336645 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.336573    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2" podUID="484e7a29-5998-440c-a9e9-8f0c50238a73" containerName="main" containerID="cri-o://c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e" gracePeriod=45
Jul 09 19:21:18.623860 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.623826    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:21:18.712360 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712322    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-home\") pod \"484e7a29-5998-440c-a9e9-8f0c50238a73\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") "
Jul 09 19:21:18.712542 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712392    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-tmp-dir\") pod \"484e7a29-5998-440c-a9e9-8f0c50238a73\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") "
Jul 09 19:21:18.712542 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712425    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-model-cache\") pod \"484e7a29-5998-440c-a9e9-8f0c50238a73\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") "
Jul 09 19:21:18.712542 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712443    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlbng\" (UniqueName: \"kubernetes.io/projected/484e7a29-5998-440c-a9e9-8f0c50238a73-kube-api-access-dlbng\") pod \"484e7a29-5998-440c-a9e9-8f0c50238a73\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") "
Jul 09 19:21:18.712542 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712482    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484e7a29-5998-440c-a9e9-8f0c50238a73-tls-certs\") pod \"484e7a29-5998-440c-a9e9-8f0c50238a73\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") "
Jul 09 19:21:18.712542 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712505    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-dshm\") pod \"484e7a29-5998-440c-a9e9-8f0c50238a73\" (UID: \"484e7a29-5998-440c-a9e9-8f0c50238a73\") "
Jul 09 19:21:18.712843 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712619    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-home" (OuterVolumeSpecName: "home") pod "484e7a29-5998-440c-a9e9-8f0c50238a73" (UID: "484e7a29-5998-440c-a9e9-8f0c50238a73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:21:18.712843 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712719    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "484e7a29-5998-440c-a9e9-8f0c50238a73" (UID: "484e7a29-5998-440c-a9e9-8f0c50238a73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:21:18.712843 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712731    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-model-cache" (OuterVolumeSpecName: "model-cache") pod "484e7a29-5998-440c-a9e9-8f0c50238a73" (UID: "484e7a29-5998-440c-a9e9-8f0c50238a73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:21:18.713022 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712879    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:21:18.713022 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712949    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:21:18.713022 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.712967    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:21:18.714666 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.714629    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484e7a29-5998-440c-a9e9-8f0c50238a73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "484e7a29-5998-440c-a9e9-8f0c50238a73" (UID: "484e7a29-5998-440c-a9e9-8f0c50238a73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:21:18.714775 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.714677    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-dshm" (OuterVolumeSpecName: "dshm") pod "484e7a29-5998-440c-a9e9-8f0c50238a73" (UID: "484e7a29-5998-440c-a9e9-8f0c50238a73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:21:18.714775 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.714695    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484e7a29-5998-440c-a9e9-8f0c50238a73-kube-api-access-dlbng" (OuterVolumeSpecName: "kube-api-access-dlbng") pod "484e7a29-5998-440c-a9e9-8f0c50238a73" (UID: "484e7a29-5998-440c-a9e9-8f0c50238a73"). InnerVolumeSpecName "kube-api-access-dlbng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:21:18.810368 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.810325    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:21:18.813782 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.813761    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbng\" (UniqueName: \"kubernetes.io/projected/484e7a29-5998-440c-a9e9-8f0c50238a73-kube-api-access-dlbng\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:21:18.813867 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.813793    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/484e7a29-5998-440c-a9e9-8f0c50238a73-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:21:18.813867 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:18.813806    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/484e7a29-5998-440c-a9e9-8f0c50238a73-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:21:19.100762 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.100730    2565 generic.go:356] "Generic (PLEG): container finished" podID="484e7a29-5998-440c-a9e9-8f0c50238a73" containerID="c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e" exitCode=0
Jul 09 19:21:19.100960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.100768    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"
Jul 09 19:21:19.100960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.100817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2" event={"ID":"484e7a29-5998-440c-a9e9-8f0c50238a73","Type":"ContainerDied","Data":"c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e"}
Jul 09 19:21:19.100960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.100866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2" event={"ID":"484e7a29-5998-440c-a9e9-8f0c50238a73","Type":"ContainerDied","Data":"2430fd6134a708ffbbe45818862391110b18a282fb95f16fbdca404d656815b9"}
Jul 09 19:21:19.100960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.100895    2565 scope.go:117] "RemoveContainer" containerID="c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e"
Jul 09 19:21:19.109444 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.109426    2565 scope.go:117] "RemoveContainer" containerID="c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e"
Jul 09 19:21:19.109725 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:21:19.109705    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e\": container with ID starting with c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e not found: ID does not exist" containerID="c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e"
Jul 09 19:21:19.109780 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.109731    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e"} err="failed to get container status \"c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e\": rpc error: code = NotFound desc = could not find container \"c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e\": container with ID starting with c089e5600809b357101e2f80f197f6294a61d100d4aa4dfc36b51c10ccd6c03e not found: ID does not exist"
Jul 09 19:21:19.118724 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.118700    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"]
Jul 09 19:21:19.122503 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.122481    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2"]
Jul 09 19:21:19.618930 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:19.618878    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5wtkh2" podUID="484e7a29-5998-440c-a9e9-8f0c50238a73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded"
Jul 09 19:21:20.868876 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:20.868841    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484e7a29-5998-440c-a9e9-8f0c50238a73" path="/var/lib/kubelet/pods/484e7a29-5998-440c-a9e9-8f0c50238a73/volumes"
Jul 09 19:21:28.810740 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:28.810688    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:21:33.010198 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:33.010166    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:21:38.810285 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:38.810235    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:21:48.810329 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:48.810281    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:21:58.810657 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:21:58.810616    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:22:08.810178 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:08.810131    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:22:18.810653 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:18.810605    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:22:28.810794 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:28.810746    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:22:38.810320 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:38.810213    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:22:48.809989 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:48.809933    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:22:56.570420 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.570383    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"]
Jul 09 19:22:56.570817 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.570741    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484e7a29-5998-440c-a9e9-8f0c50238a73" containerName="main"
Jul 09 19:22:56.570817 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.570754    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="484e7a29-5998-440c-a9e9-8f0c50238a73" containerName="main"
Jul 09 19:22:56.570817 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.570772    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af354ac7-d71a-4744-8a24-cfefce87dccb" containerName="main"
Jul 09 19:22:56.570817 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.570778    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af354ac7-d71a-4744-8a24-cfefce87dccb" containerName="main"
Jul 09 19:22:56.570975 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.570829    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="af354ac7-d71a-4744-8a24-cfefce87dccb" containerName="main"
Jul 09 19:22:56.570975 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.570838    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="484e7a29-5998-440c-a9e9-8f0c50238a73" containerName="main"
Jul 09 19:22:56.573794 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.573777    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.576624 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.576601    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 09 19:22:56.586383 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.586358    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"]
Jul 09 19:22:56.622171 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.622132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.622171 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.622176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.622390 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.622218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.622390 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.622289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfqk\" (UniqueName: \"kubernetes.io/projected/43b368bf-646b-4910-af62-b0e3f3b53be0-kube-api-access-6sfqk\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.622390 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.622320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.622390 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.622339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b368bf-646b-4910-af62-b0e3f3b53be0-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.722915 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.722868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfqk\" (UniqueName: \"kubernetes.io/projected/43b368bf-646b-4910-af62-b0e3f3b53be0-kube-api-access-6sfqk\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.723089 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.722925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.723089 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.722945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b368bf-646b-4910-af62-b0e3f3b53be0-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.723089 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.723022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.723089 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.723060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.723310 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.723086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.723372 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.723344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.723488 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.723466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.723549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.723472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.725297 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.725276    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.725456 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.725440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b368bf-646b-4910-af62-b0e3f3b53be0-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.734472 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.734447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfqk\" (UniqueName: \"kubernetes.io/projected/43b368bf-646b-4910-af62-b0e3f3b53be0-kube-api-access-6sfqk\") pod \"rolling-upgrade-test-kserve-859df57bcb-pmcmd\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:56.883887 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:56.883807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:57.021163 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:57.020975    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"]
Jul 09 19:22:57.023446 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:22:57.023421    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b368bf_646b_4910_af62_b0e3f3b53be0.slice/crio-5ba16a9f2e5161d3ddca6e8c999d5b7143e9af48ce9b81c01d6864477ee6f68c WatchSource:0}: Error finding container 5ba16a9f2e5161d3ddca6e8c999d5b7143e9af48ce9b81c01d6864477ee6f68c: Status 404 returned error can't find the container with id 5ba16a9f2e5161d3ddca6e8c999d5b7143e9af48ce9b81c01d6864477ee6f68c
Jul 09 19:22:57.574328 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:57.574293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd" event={"ID":"43b368bf-646b-4910-af62-b0e3f3b53be0","Type":"ContainerStarted","Data":"76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a"}
Jul 09 19:22:57.574713 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:57.574347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd" event={"ID":"43b368bf-646b-4910-af62-b0e3f3b53be0","Type":"ContainerStarted","Data":"5ba16a9f2e5161d3ddca6e8c999d5b7143e9af48ce9b81c01d6864477ee6f68c"}
Jul 09 19:22:57.593112 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:57.593056    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd" podStartSLOduration=1.5930375890000001 podStartE2EDuration="1.593037589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:22:57.592607674 +0000 UTC m=+4697.310870440" watchObservedRunningTime="2026-07-09 19:22:57.593037589 +0000 UTC m=+4697.311300357"
Jul 09 19:22:57.884963 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:57.884843    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:22:58.820285 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:58.820257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:22:58.828203 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:22:58.828174    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:23:06.884280 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:06.884239    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:23:06.899220 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:06.899191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:23:07.633102 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:07.633069    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:23:24.699207 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:24.699166    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"]
Jul 09 19:23:39.719642 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:39.719592    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd" podUID="43b368bf-646b-4910-af62-b0e3f3b53be0" containerName="main" containerID="cri-o://76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a" gracePeriod=45
Jul 09 19:23:39.973188 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:39.973121    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:23:40.036182 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-model-cache\") pod \"43b368bf-646b-4910-af62-b0e3f3b53be0\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") "
Jul 09 19:23:40.036364 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036241    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-dshm\") pod \"43b368bf-646b-4910-af62-b0e3f3b53be0\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") "
Jul 09 19:23:40.036364 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036273    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-home\") pod \"43b368bf-646b-4910-af62-b0e3f3b53be0\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") "
Jul 09 19:23:40.036364 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036339    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-tmp-dir\") pod \"43b368bf-646b-4910-af62-b0e3f3b53be0\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") "
Jul 09 19:23:40.036541 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036364    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sfqk\" (UniqueName: \"kubernetes.io/projected/43b368bf-646b-4910-af62-b0e3f3b53be0-kube-api-access-6sfqk\") pod \"43b368bf-646b-4910-af62-b0e3f3b53be0\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") "
Jul 09 19:23:40.036541 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036394    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b368bf-646b-4910-af62-b0e3f3b53be0-tls-certs\") pod \"43b368bf-646b-4910-af62-b0e3f3b53be0\" (UID: \"43b368bf-646b-4910-af62-b0e3f3b53be0\") "
Jul 09 19:23:40.036541 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036457    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-model-cache" (OuterVolumeSpecName: "model-cache") pod "43b368bf-646b-4910-af62-b0e3f3b53be0" (UID: "43b368bf-646b-4910-af62-b0e3f3b53be0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:23:40.036693 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036641    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43b368bf-646b-4910-af62-b0e3f3b53be0" (UID: "43b368bf-646b-4910-af62-b0e3f3b53be0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:23:40.036693 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036650    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-home" (OuterVolumeSpecName: "home") pod "43b368bf-646b-4910-af62-b0e3f3b53be0" (UID: "43b368bf-646b-4910-af62-b0e3f3b53be0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:23:40.036783 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.036717    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:23:40.038353 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.038330    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-dshm" (OuterVolumeSpecName: "dshm") pod "43b368bf-646b-4910-af62-b0e3f3b53be0" (UID: "43b368bf-646b-4910-af62-b0e3f3b53be0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:23:40.038548 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.038516    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b368bf-646b-4910-af62-b0e3f3b53be0-kube-api-access-6sfqk" (OuterVolumeSpecName: "kube-api-access-6sfqk") pod "43b368bf-646b-4910-af62-b0e3f3b53be0" (UID: "43b368bf-646b-4910-af62-b0e3f3b53be0"). InnerVolumeSpecName "kube-api-access-6sfqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:23:40.038642 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.038580    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b368bf-646b-4910-af62-b0e3f3b53be0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43b368bf-646b-4910-af62-b0e3f3b53be0" (UID: "43b368bf-646b-4910-af62-b0e3f3b53be0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:23:40.137490 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.137454    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:23:40.137490 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.137484    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:23:40.137490 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.137495    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfqk\" (UniqueName: \"kubernetes.io/projected/43b368bf-646b-4910-af62-b0e3f3b53be0-kube-api-access-6sfqk\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:23:40.137736 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.137509    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b368bf-646b-4910-af62-b0e3f3b53be0-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:23:40.137736 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.137517    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b368bf-646b-4910-af62-b0e3f3b53be0-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:23:40.775427 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.775390    2565 generic.go:356] "Generic (PLEG): container finished" podID="43b368bf-646b-4910-af62-b0e3f3b53be0" containerID="76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a" exitCode=0
Jul 09 19:23:40.775947 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.775439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd" event={"ID":"43b368bf-646b-4910-af62-b0e3f3b53be0","Type":"ContainerDied","Data":"76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a"}
Jul 09 19:23:40.775947 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.775445    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"
Jul 09 19:23:40.775947 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.775512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd" event={"ID":"43b368bf-646b-4910-af62-b0e3f3b53be0","Type":"ContainerDied","Data":"5ba16a9f2e5161d3ddca6e8c999d5b7143e9af48ce9b81c01d6864477ee6f68c"}
Jul 09 19:23:40.775947 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.775536    2565 scope.go:117] "RemoveContainer" containerID="76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a"
Jul 09 19:23:40.784535 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.784479    2565 scope.go:117] "RemoveContainer" containerID="76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a"
Jul 09 19:23:40.784808 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:23:40.784784    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a\": container with ID starting with 76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a not found: ID does not exist" containerID="76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a"
Jul 09 19:23:40.784935 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.784814    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a"} err="failed to get container status \"76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a\": rpc error: code = NotFound desc = could not find container \"76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a\": container with ID starting with 76ae0f92fe536dc2731e2fb008c52930aa35e6e2793b02e5cf21a9413178890a not found: ID does not exist"
Jul 09 19:23:40.798132 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.798104    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"]
Jul 09 19:23:40.802323 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.802299    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-pmcmd"]
Jul 09 19:23:40.868725 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:23:40.868696    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43b368bf-646b-4910-af62-b0e3f3b53be0" path="/var/lib/kubelet/pods/43b368bf-646b-4910-af62-b0e3f3b53be0/volumes"
Jul 09 19:24:40.270118 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.270077    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"]
Jul 09 19:24:40.281287 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:24:40.281250    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:24:40.281287 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:24:40.281287 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:24:40.281287 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17" cmd=["/bin/sleep","15"]
Jul 09 19:24:40.281624 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.281305    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" message="" err=<
Jul 09 19:24:40.281624 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:24:40.281624 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:24:40.281624 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:24:40.281624 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:24:40.281318    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:24:40.281624 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:24:40.281624 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:24:40.281624 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" podUID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" containerName="main" containerID="cri-o://61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17"
Jul 09 19:24:40.281624 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.281353    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" podUID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" containerName="main" containerID="cri-o://61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17" gracePeriod=60
Jul 09 19:24:40.283311 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.283287    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"]
Jul 09 19:24:40.541869 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.541844    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:24:40.686312 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.686285    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb7tq\" (UniqueName: \"kubernetes.io/projected/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kube-api-access-mb7tq\") pod \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") "
Jul 09 19:24:40.686491 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.686346    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tls-certs\") pod \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") "
Jul 09 19:24:40.686491 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.686397    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tokenizer-uds\") pod \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") "
Jul 09 19:24:40.686491 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.686420    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kserve-provision-location\") pod \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\" (UID: \"adcb4019-0f5f-48f4-8624-b5a645c3bec2\") "
Jul 09 19:24:40.686682 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.686653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "adcb4019-0f5f-48f4-8624-b5a645c3bec2" (UID: "adcb4019-0f5f-48f4-8624-b5a645c3bec2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:24:40.687122 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.687095    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "adcb4019-0f5f-48f4-8624-b5a645c3bec2" (UID: "adcb4019-0f5f-48f4-8624-b5a645c3bec2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:24:40.688329 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.688310    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kube-api-access-mb7tq" (OuterVolumeSpecName: "kube-api-access-mb7tq") pod "adcb4019-0f5f-48f4-8624-b5a645c3bec2" (UID: "adcb4019-0f5f-48f4-8624-b5a645c3bec2"). InnerVolumeSpecName "kube-api-access-mb7tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:24:40.688446 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.688426    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adcb4019-0f5f-48f4-8624-b5a645c3bec2" (UID: "adcb4019-0f5f-48f4-8624-b5a645c3bec2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:24:40.787649 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.787570    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:40.787649 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.787600    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:40.787649 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.787611    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb7tq\" (UniqueName: \"kubernetes.io/projected/adcb4019-0f5f-48f4-8624-b5a645c3bec2-kube-api-access-mb7tq\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:40.787649 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:40.787620    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcb4019-0f5f-48f4-8624-b5a645c3bec2-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:41.066399 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.066322    2565 generic.go:356] "Generic (PLEG): container finished" podID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" containerID="61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17" exitCode=0
Jul 09 19:24:41.066399 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.066371    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"
Jul 09 19:24:41.066566 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.066405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" event={"ID":"adcb4019-0f5f-48f4-8624-b5a645c3bec2","Type":"ContainerDied","Data":"61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17"}
Jul 09 19:24:41.066566 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.066466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w" event={"ID":"adcb4019-0f5f-48f4-8624-b5a645c3bec2","Type":"ContainerDied","Data":"161a42fae890f52bd12bda492dcbacb6396bb857c6b4fbf2cc3f1950771d9775"}
Jul 09 19:24:41.066566 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.066491    2565 scope.go:117] "RemoveContainer" containerID="61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17"
Jul 09 19:24:41.079167 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.078681    2565 scope.go:117] "RemoveContainer" containerID="3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636"
Jul 09 19:24:41.087849 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.087827    2565 scope.go:117] "RemoveContainer" containerID="61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17"
Jul 09 19:24:41.088105 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:24:41.088085    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17\": container with ID starting with 61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17 not found: ID does not exist" containerID="61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17"
Jul 09 19:24:41.088178 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.088116    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17"} err="failed to get container status \"61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17\": rpc error: code = NotFound desc = could not find container \"61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17\": container with ID starting with 61023089662f51f7ba0c2371944e6acde5f6f4e9eb92d5ab7eade37696540d17 not found: ID does not exist"
Jul 09 19:24:41.088178 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.088140    2565 scope.go:117] "RemoveContainer" containerID="3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636"
Jul 09 19:24:41.088480 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:24:41.088448    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636\": container with ID starting with 3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636 not found: ID does not exist" containerID="3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636"
Jul 09 19:24:41.088574 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.088479    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636"} err="failed to get container status \"3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636\": rpc error: code = NotFound desc = could not find container \"3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636\": container with ID starting with 3bf591141548048e975229db8a59acf9657a32519ca0a35370fe1374120ee636 not found: ID does not exist"
Jul 09 19:24:41.091516 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.091494    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"]
Jul 09 19:24:41.098107 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.098085    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-86c87d5cd9-xkg8w"]
Jul 09 19:24:41.102729 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.102705    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:24:41.110065 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:41.110049    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:24:42.868591 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:42.868546    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" path="/var/lib/kubelet/pods/adcb4019-0f5f-48f4-8624-b5a645c3bec2/volumes"
Jul 09 19:24:55.308024 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:55.307975    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" containerID="cri-o://ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2" gracePeriod=45
Jul 09 19:24:55.820892 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:55.820846    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 09 19:24:56.088688 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.088664    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:24:56.138585 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.138551    2565 generic.go:356] "Generic (PLEG): container finished" podID="7da907bc-769e-461c-9b25-e6f09c204add" containerID="ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2" exitCode=0
Jul 09 19:24:56.138766 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.138606    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"
Jul 09 19:24:56.138766 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.138633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" event={"ID":"7da907bc-769e-461c-9b25-e6f09c204add","Type":"ContainerDied","Data":"ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2"}
Jul 09 19:24:56.138766 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.138689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc" event={"ID":"7da907bc-769e-461c-9b25-e6f09c204add","Type":"ContainerDied","Data":"04654ed49820065a471b03dae0b152af242834c468a5378fe6e508cda8e1455b"}
Jul 09 19:24:56.138766 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.138712    2565 scope.go:117] "RemoveContainer" containerID="ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2"
Jul 09 19:24:56.146923 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.146893    2565 scope.go:117] "RemoveContainer" containerID="9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43"
Jul 09 19:24:56.154452 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.154437    2565 scope.go:117] "RemoveContainer" containerID="ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2"
Jul 09 19:24:56.154697 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:24:56.154677    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2\": container with ID starting with ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2 not found: ID does not exist" containerID="ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2"
Jul 09 19:24:56.154752 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.154706    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2"} err="failed to get container status \"ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2\": rpc error: code = NotFound desc = could not find container \"ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2\": container with ID starting with ad0951e5c90c0cd5fc700870eafa09cf36f7407d0d94f6b7f13da8ab53bf89b2 not found: ID does not exist"
Jul 09 19:24:56.154752 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.154722    2565 scope.go:117] "RemoveContainer" containerID="9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43"
Jul 09 19:24:56.154981 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:24:56.154929    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43\": container with ID starting with 9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43 not found: ID does not exist" containerID="9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43"
Jul 09 19:24:56.154981 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.154956    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43"} err="failed to get container status \"9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43\": rpc error: code = NotFound desc = could not find container \"9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43\": container with ID starting with 9adaef9fed9fe25f85e6f849d66edaa046029966d18fb78b959c770d704b6f43 not found: ID does not exist"
Jul 09 19:24:56.230947 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.230896    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da907bc-769e-461c-9b25-e6f09c204add-tls-certs\") pod \"7da907bc-769e-461c-9b25-e6f09c204add\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") "
Jul 09 19:24:56.230947 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.230953    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-dshm\") pod \"7da907bc-769e-461c-9b25-e6f09c204add\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") "
Jul 09 19:24:56.231182 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.231015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-tmp-dir\") pod \"7da907bc-769e-461c-9b25-e6f09c204add\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") "
Jul 09 19:24:56.231182 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.231046    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-model-cache\") pod \"7da907bc-769e-461c-9b25-e6f09c204add\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") "
Jul 09 19:24:56.231182 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.231076    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7mnt\" (UniqueName: \"kubernetes.io/projected/7da907bc-769e-461c-9b25-e6f09c204add-kube-api-access-n7mnt\") pod \"7da907bc-769e-461c-9b25-e6f09c204add\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") "
Jul 09 19:24:56.231182 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.231128    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-kserve-provision-location\") pod \"7da907bc-769e-461c-9b25-e6f09c204add\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") "
Jul 09 19:24:56.231182 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.231150    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-home\") pod \"7da907bc-769e-461c-9b25-e6f09c204add\" (UID: \"7da907bc-769e-461c-9b25-e6f09c204add\") "
Jul 09 19:24:56.231447 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.231310    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-model-cache" (OuterVolumeSpecName: "model-cache") pod "7da907bc-769e-461c-9b25-e6f09c204add" (UID: "7da907bc-769e-461c-9b25-e6f09c204add"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:24:56.231447 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.231442    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:56.231861 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.231836    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-home" (OuterVolumeSpecName: "home") pod "7da907bc-769e-461c-9b25-e6f09c204add" (UID: "7da907bc-769e-461c-9b25-e6f09c204add"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:24:56.233379 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.233345    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da907bc-769e-461c-9b25-e6f09c204add-kube-api-access-n7mnt" (OuterVolumeSpecName: "kube-api-access-n7mnt") pod "7da907bc-769e-461c-9b25-e6f09c204add" (UID: "7da907bc-769e-461c-9b25-e6f09c204add"). InnerVolumeSpecName "kube-api-access-n7mnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:24:56.233543 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.233518    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-dshm" (OuterVolumeSpecName: "dshm") pod "7da907bc-769e-461c-9b25-e6f09c204add" (UID: "7da907bc-769e-461c-9b25-e6f09c204add"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:24:56.233543 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.233534    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7da907bc-769e-461c-9b25-e6f09c204add-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7da907bc-769e-461c-9b25-e6f09c204add" (UID: "7da907bc-769e-461c-9b25-e6f09c204add"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:24:56.243260 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.243230    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7da907bc-769e-461c-9b25-e6f09c204add" (UID: "7da907bc-769e-461c-9b25-e6f09c204add"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:24:56.298270 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.298226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7da907bc-769e-461c-9b25-e6f09c204add" (UID: "7da907bc-769e-461c-9b25-e6f09c204add"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:24:56.332779 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.332718    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7da907bc-769e-461c-9b25-e6f09c204add-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:56.332779 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.332746    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:56.332779 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.332757    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:56.332779 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.332765    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mnt\" (UniqueName: \"kubernetes.io/projected/7da907bc-769e-461c-9b25-e6f09c204add-kube-api-access-n7mnt\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:56.332779 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.332775    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:56.333232 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.332784    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7da907bc-769e-461c-9b25-e6f09c204add-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:24:56.462445 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.462416    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"]
Jul 09 19:24:56.465885 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.465859    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-59fd59cf4d-r5fjc"]
Jul 09 19:24:56.868372 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:24:56.868336    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7da907bc-769e-461c-9b25-e6f09c204add" path="/var/lib/kubelet/pods/7da907bc-769e-461c-9b25-e6f09c204add/volumes"
Jul 09 19:25:25.486657 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.486624    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"]
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.486961    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" containerName="storage-initializer"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.486975    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" containerName="storage-initializer"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.486985    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="storage-initializer"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.486992    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="storage-initializer"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.486999    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" containerName="main"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.487004    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" containerName="main"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.487016    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.487021    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.487033    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b368bf-646b-4910-af62-b0e3f3b53be0" containerName="main"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.487038    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b368bf-646b-4910-af62-b0e3f3b53be0" containerName="main"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.487107    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da907bc-769e-461c-9b25-e6f09c204add" containerName="main"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.487115    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b368bf-646b-4910-af62-b0e3f3b53be0" containerName="main"
Jul 09 19:25:25.487165 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.487124    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="adcb4019-0f5f-48f4-8624-b5a645c3bec2" containerName="main"
Jul 09 19:25:25.490085 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.490066    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.492571 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.492546    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 09 19:25:25.492724 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.492570    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 19:25:25.507266 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.505839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"]
Jul 09 19:25:25.584051 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.584012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-dshm\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.584051 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.584052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-model-cache\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.584262 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.584106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-home\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.584262 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.584126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tls-certs\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.584262 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.584147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kserve-provision-location\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.584262 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.584177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6zq5\" (UniqueName: \"kubernetes.io/projected/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kube-api-access-t6zq5\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.584396 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.584292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tmp-dir\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.685799 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.685756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tmp-dir\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686017 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.685829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-dshm\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686017 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.685847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-model-cache\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686017 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.685992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-home\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686195 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.686025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tls-certs\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686195 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.686052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kserve-provision-location\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686195 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.686085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6zq5\" (UniqueName: \"kubernetes.io/projected/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kube-api-access-t6zq5\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686357 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.686295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tmp-dir\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686357 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.686316    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-model-cache\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686474 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.686387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kserve-provision-location\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.686474 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.686449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-home\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.688071 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.688048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-dshm\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.688516 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.688494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tls-certs\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.694777 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.694751    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6zq5\" (UniqueName: \"kubernetes.io/projected/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kube-api-access-t6zq5\") pod \"prestop-hook-test-kserve-5545457c4d-m6q58\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.737961 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.737868    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"]
Jul 09 19:25:25.741701 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.741681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.744611 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.744592    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-2ksnd\""
Jul 09 19:25:25.753830 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.753805    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"]
Jul 09 19:25:25.787271 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.787239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10d8176-2e1f-4640-b3fd-637dd506591d-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.787469 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.787291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzd8g\" (UniqueName: \"kubernetes.io/projected/f10d8176-2e1f-4640-b3fd-637dd506591d-kube-api-access-gzd8g\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.787469 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.787311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.787469 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.787431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.800717 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.800698    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:25.888409 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.888371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzd8g\" (UniqueName: \"kubernetes.io/projected/f10d8176-2e1f-4640-b3fd-637dd506591d-kube-api-access-gzd8g\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.888549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.888426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.888549 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.888525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.888668 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.888573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10d8176-2e1f-4640-b3fd-637dd506591d-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.889377 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.889343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.889611 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.889588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.891884 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.891855    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10d8176-2e1f-4640-b3fd-637dd506591d-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.898536 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.898514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzd8g\" (UniqueName: \"kubernetes.io/projected/f10d8176-2e1f-4640-b3fd-637dd506591d-kube-api-access-gzd8g\") pod \"prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:25.945163 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:25.945121    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"]
Jul 09 19:25:25.948867 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:25:25.948836    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da0925f_10ca_494b_bb60_79cbb9e0a13e.slice/crio-c52a28ff233173a43d2192fcb24b8a1f26c2d7ad19be9f3d868c633b55e8e0ad WatchSource:0}: Error finding container c52a28ff233173a43d2192fcb24b8a1f26c2d7ad19be9f3d868c633b55e8e0ad: Status 404 returned error can't find the container with id c52a28ff233173a43d2192fcb24b8a1f26c2d7ad19be9f3d868c633b55e8e0ad
Jul 09 19:25:26.052232 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:26.052194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:26.187947 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:26.187845    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"]
Jul 09 19:25:26.190310 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:25:26.190284    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10d8176_2e1f_4640_b3fd_637dd506591d.slice/crio-66727e11678ddad63cfeac35f320633b15a48949cb3e52ec3cce8f8bec7a3227 WatchSource:0}: Error finding container 66727e11678ddad63cfeac35f320633b15a48949cb3e52ec3cce8f8bec7a3227: Status 404 returned error can't find the container with id 66727e11678ddad63cfeac35f320633b15a48949cb3e52ec3cce8f8bec7a3227
Jul 09 19:25:26.274223 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:26.274179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" event={"ID":"f10d8176-2e1f-4640-b3fd-637dd506591d","Type":"ContainerStarted","Data":"9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92"}
Jul 09 19:25:26.274420 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:26.274252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" event={"ID":"f10d8176-2e1f-4640-b3fd-637dd506591d","Type":"ContainerStarted","Data":"66727e11678ddad63cfeac35f320633b15a48949cb3e52ec3cce8f8bec7a3227"}
Jul 09 19:25:26.275847 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:26.275809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" event={"ID":"0da0925f-10ca-494b-bb60-79cbb9e0a13e","Type":"ContainerStarted","Data":"43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335"}
Jul 09 19:25:26.275981 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:26.275869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" event={"ID":"0da0925f-10ca-494b-bb60-79cbb9e0a13e","Type":"ContainerStarted","Data":"c52a28ff233173a43d2192fcb24b8a1f26c2d7ad19be9f3d868c633b55e8e0ad"}
Jul 09 19:25:27.282974 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:27.282871    2565 generic.go:356] "Generic (PLEG): container finished" podID="f10d8176-2e1f-4640-b3fd-637dd506591d" containerID="9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92" exitCode=0
Jul 09 19:25:27.283344 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:27.282972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" event={"ID":"f10d8176-2e1f-4640-b3fd-637dd506591d","Type":"ContainerDied","Data":"9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92"}
Jul 09 19:25:28.292833 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:28.292801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" event={"ID":"f10d8176-2e1f-4640-b3fd-637dd506591d","Type":"ContainerStarted","Data":"cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4"}
Jul 09 19:25:28.293877 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:28.292929    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:25:28.317635 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:28.317568    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" podStartSLOduration=2.56416948 podStartE2EDuration="3.317536111s" totalImagesPullingTime="0s" totalInitContainerRuntime="753.366631ms" isStatefulPod=false podCreationTimestamp="2026-07-09 19:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:25:28.314240966 +0000 UTC m=+4848.032503730" watchObservedRunningTime="2026-07-09 19:25:28.317536111 +0000 UTC m=+4848.035798871"
Jul 09 19:25:30.304605 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:30.304574    2565 generic.go:356] "Generic (PLEG): container finished" podID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerID="43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335" exitCode=0
Jul 09 19:25:30.305107 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:30.304653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" event={"ID":"0da0925f-10ca-494b-bb60-79cbb9e0a13e","Type":"ContainerDied","Data":"43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335"}
Jul 09 19:25:31.311351 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:31.311306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" event={"ID":"0da0925f-10ca-494b-bb60-79cbb9e0a13e","Type":"ContainerStarted","Data":"a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1"}
Jul 09 19:25:31.336887 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:31.336769    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podStartSLOduration=2.150458177 podStartE2EDuration="6.336751497s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.18629332s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:25:31.333955532 +0000 UTC m=+4851.052218297" watchObservedRunningTime="2026-07-09 19:25:31.336751497 +0000 UTC m=+4851.055014260"
Jul 09 19:25:31.800826 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:31.800781    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:35.801278 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:35.801232    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:25:35.803001 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:35.802970    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:25:45.802013 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:45.801958    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:25:55.801984 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:55.801941    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:25:59.300111 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:25:59.300072    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:26:05.802116 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:26:05.802063    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:26:15.802128 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:26:15.802080    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:26:25.802124 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:26:25.802081    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:26:35.802172 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:26:35.802119    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:26:45.801618 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:26:45.801565    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:26:55.802070 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:26:55.802021    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:27:05.802177 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:05.802065    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:27:15.801570 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:15.801516    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:27:25.811266 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:25.811235    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:27:25.820049 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:25.820009    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:27:30.164141 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.164106    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"]
Jul 09 19:27:30.197077 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.197043    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"]
Jul 09 19:27:30.202135 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.202113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.213297 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.213272    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"]
Jul 09 19:27:30.215359 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.215334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-home\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.215497 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.215449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ffpm\" (UniqueName: \"kubernetes.io/projected/2152b310-5619-4ef7-a96d-a3522d3a9afd-kube-api-access-9ffpm\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.215560 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.215522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-tmp-dir\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.215618 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.215564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2152b310-5619-4ef7-a96d-a3522d3a9afd-tls-certs\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.215676 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.215631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-dshm\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.215735 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.215677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-kserve-provision-location\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.215788 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.215736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-model-cache\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.316665 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.316626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ffpm\" (UniqueName: \"kubernetes.io/projected/2152b310-5619-4ef7-a96d-a3522d3a9afd-kube-api-access-9ffpm\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.316865 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.316678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-tmp-dir\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.316865 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.316697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2152b310-5619-4ef7-a96d-a3522d3a9afd-tls-certs\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.316865 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.316731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-dshm\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.316865 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.316756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-kserve-provision-location\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.316865 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.316779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-model-cache\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.316865 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.316809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-home\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.317207 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.317131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-tmp-dir\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.317267 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.317200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-kserve-provision-location\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.317267 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.317231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-home\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.317374 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.317331    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-model-cache\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.319097 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.319078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-dshm\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.319279 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.319260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2152b310-5619-4ef7-a96d-a3522d3a9afd-tls-certs\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.325837 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.325808    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ffpm\" (UniqueName: \"kubernetes.io/projected/2152b310-5619-4ef7-a96d-a3522d3a9afd-kube-api-access-9ffpm\") pod \"prestop-hook-test-kserve-5545457c4d-krxfb\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.512504 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.512463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:30.656980 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.656953    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"]
Jul 09 19:27:30.659187 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:27:30.659159    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2152b310_5619_4ef7_a96d_a3522d3a9afd.slice/crio-d8ba8ba84b9cc0dce75898c5304a494fcd675e525d944203693b74870bde5642 WatchSource:0}: Error finding container d8ba8ba84b9cc0dce75898c5304a494fcd675e525d944203693b74870bde5642: Status 404 returned error can't find the container with id d8ba8ba84b9cc0dce75898c5304a494fcd675e525d944203693b74870bde5642
Jul 09 19:27:30.890868 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.890778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" event={"ID":"2152b310-5619-4ef7-a96d-a3522d3a9afd","Type":"ContainerStarted","Data":"d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776"}
Jul 09 19:27:30.890868 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:30.890829    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" event={"ID":"2152b310-5619-4ef7-a96d-a3522d3a9afd","Type":"ContainerStarted","Data":"d8ba8ba84b9cc0dce75898c5304a494fcd675e525d944203693b74870bde5642"}
Jul 09 19:27:34.924033 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:34.924001    2565 generic.go:356] "Generic (PLEG): container finished" podID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerID="d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776" exitCode=0
Jul 09 19:27:34.924397 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:34.924074    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" event={"ID":"2152b310-5619-4ef7-a96d-a3522d3a9afd","Type":"ContainerDied","Data":"d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776"}
Jul 09 19:27:35.930812 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:35.930778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" event={"ID":"2152b310-5619-4ef7-a96d-a3522d3a9afd","Type":"ContainerStarted","Data":"5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01"}
Jul 09 19:27:35.957113 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:35.957048    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" podStartSLOduration=2.549880489 podStartE2EDuration="5.957025452s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.407144963s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:27:35.954782939 +0000 UTC m=+4975.673045705" watchObservedRunningTime="2026-07-09 19:27:35.957025452 +0000 UTC m=+4975.675288219"
Jul 09 19:27:36.513077 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:36.513034    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:40.512628 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:40.512586    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:27:40.514395 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:40.514365    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 09 19:27:45.187355 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:45.187304    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" containerID="cri-o://a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1" gracePeriod=45
Jul 09 19:27:45.812525 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:45.812468    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 09 19:27:45.971821 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:45.971746    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:27:45.982968 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:45.982939    2565 generic.go:356] "Generic (PLEG): container finished" podID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerID="a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1" exitCode=0
Jul 09 19:27:45.983121 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:45.982999    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"
Jul 09 19:27:45.983121 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:45.983009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" event={"ID":"0da0925f-10ca-494b-bb60-79cbb9e0a13e","Type":"ContainerDied","Data":"a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1"}
Jul 09 19:27:45.983121 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:45.983048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58" event={"ID":"0da0925f-10ca-494b-bb60-79cbb9e0a13e","Type":"ContainerDied","Data":"c52a28ff233173a43d2192fcb24b8a1f26c2d7ad19be9f3d868c633b55e8e0ad"}
Jul 09 19:27:45.983121 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:45.983069    2565 scope.go:117] "RemoveContainer" containerID="a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1"
Jul 09 19:27:45.991205 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:45.991168    2565 scope.go:117] "RemoveContainer" containerID="43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335"
Jul 09 19:27:46.001439 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.001414    2565 scope.go:117] "RemoveContainer" containerID="a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1"
Jul 09 19:27:46.001778 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:27:46.001753    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1\": container with ID starting with a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1 not found: ID does not exist" containerID="a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1"
Jul 09 19:27:46.001894 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.001785    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1"} err="failed to get container status \"a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1\": rpc error: code = NotFound desc = could not find container \"a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1\": container with ID starting with a687fd3c885ca051a25aa19b3f187f7ea00b9a36dc31fb7c95d358a501bf57f1 not found: ID does not exist"
Jul 09 19:27:46.001894 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.001807    2565 scope.go:117] "RemoveContainer" containerID="43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335"
Jul 09 19:27:46.002155 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:27:46.002129    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335\": container with ID starting with 43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335 not found: ID does not exist" containerID="43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335"
Jul 09 19:27:46.002308 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.002164    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335"} err="failed to get container status \"43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335\": rpc error: code = NotFound desc = could not find container \"43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335\": container with ID starting with 43eea91e930da9f38c6cdc23543fca9cf5471cdb2366d36ee67a60304ce04335 not found: ID does not exist"
Jul 09 19:27:46.063081 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.063040    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kserve-provision-location\") pod \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") "
Jul 09 19:27:46.063081 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.063079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-model-cache\") pod \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") "
Jul 09 19:27:46.063355 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.063113    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-dshm\") pod \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") "
Jul 09 19:27:46.063355 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.063155    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-home\") pod \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") "
Jul 09 19:27:46.063355 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.063178    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tls-certs\") pod \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") "
Jul 09 19:27:46.063355 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.063234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tmp-dir\") pod \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") "
Jul 09 19:27:46.063355 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.063270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6zq5\" (UniqueName: \"kubernetes.io/projected/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kube-api-access-t6zq5\") pod \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\" (UID: \"0da0925f-10ca-494b-bb60-79cbb9e0a13e\") "
Jul 09 19:27:46.063894 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.063859    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-home" (OuterVolumeSpecName: "home") pod "0da0925f-10ca-494b-bb60-79cbb9e0a13e" (UID: "0da0925f-10ca-494b-bb60-79cbb9e0a13e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:46.064037 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.063930    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0da0925f-10ca-494b-bb60-79cbb9e0a13e" (UID: "0da0925f-10ca-494b-bb60-79cbb9e0a13e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:46.065361 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.065337    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-dshm" (OuterVolumeSpecName: "dshm") pod "0da0925f-10ca-494b-bb60-79cbb9e0a13e" (UID: "0da0925f-10ca-494b-bb60-79cbb9e0a13e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:46.065776 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.065756    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0da0925f-10ca-494b-bb60-79cbb9e0a13e" (UID: "0da0925f-10ca-494b-bb60-79cbb9e0a13e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:27:46.066234 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.066210    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kube-api-access-t6zq5" (OuterVolumeSpecName: "kube-api-access-t6zq5") pod "0da0925f-10ca-494b-bb60-79cbb9e0a13e" (UID: "0da0925f-10ca-494b-bb60-79cbb9e0a13e"). InnerVolumeSpecName "kube-api-access-t6zq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:27:46.081125 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.081082    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0da0925f-10ca-494b-bb60-79cbb9e0a13e" (UID: "0da0925f-10ca-494b-bb60-79cbb9e0a13e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:46.122706 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.122665    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0da0925f-10ca-494b-bb60-79cbb9e0a13e" (UID: "0da0925f-10ca-494b-bb60-79cbb9e0a13e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:46.164126 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.164083    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:27:46.164126 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.164118    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:27:46.164126 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.164132    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:27:46.164374 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.164144    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6zq5\" (UniqueName: \"kubernetes.io/projected/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kube-api-access-t6zq5\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:27:46.164374 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.164158    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:27:46.164374 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.164169    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:27:46.164374 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.164179    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da0925f-10ca-494b-bb60-79cbb9e0a13e-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:27:46.306500 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.306465    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"]
Jul 09 19:27:46.312307 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.312282    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-m6q58"]
Jul 09 19:27:46.868251 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:46.868219    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" path="/var/lib/kubelet/pods/0da0925f-10ca-494b-bb60-79cbb9e0a13e/volumes"
Jul 09 19:27:50.513477 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:27:50.513432    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 09 19:28:00.512923 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:00.512868    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Jul 09 19:28:01.484512 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.484473    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"]
Jul 09 19:28:01.492272 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.492237    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"]
Jul 09 19:28:01.501130 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:28:01.501086    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:28:01.501130 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:28:01.501130 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:28:01.501130 ip-10-0-129-87 kubenswrapper[2565]:  > containerID="cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4" cmd=["/bin/sleep","15"]
Jul 09 19:28:01.501479 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.501150    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" message="" err=<
Jul 09 19:28:01.501479 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:28:01.501479 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:28:01.501479 ip-10-0-129-87 kubenswrapper[2565]:  >
Jul 09 19:28:01.501479 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:28:01.501168    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:28:01.501479 ip-10-0-129-87 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:28:01.501479 ip-10-0-129-87 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 09 19:28:01.501479 ip-10-0-129-87 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" podUID="f10d8176-2e1f-4640-b3fd-637dd506591d" containerName="main" containerID="cri-o://cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4"
Jul 09 19:28:01.501479 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.501215    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" podUID="f10d8176-2e1f-4640-b3fd-637dd506591d" containerName="main" containerID="cri-o://cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4" gracePeriod=60
Jul 09 19:28:01.764423 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.764393    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:28:01.910750 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.910712    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10d8176-2e1f-4640-b3fd-637dd506591d-tls-certs\") pod \"f10d8176-2e1f-4640-b3fd-637dd506591d\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") "
Jul 09 19:28:01.910750 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.910752    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-kserve-provision-location\") pod \"f10d8176-2e1f-4640-b3fd-637dd506591d\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") "
Jul 09 19:28:01.911055 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.910792    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-tokenizer-uds\") pod \"f10d8176-2e1f-4640-b3fd-637dd506591d\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") "
Jul 09 19:28:01.911055 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.910843    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzd8g\" (UniqueName: \"kubernetes.io/projected/f10d8176-2e1f-4640-b3fd-637dd506591d-kube-api-access-gzd8g\") pod \"f10d8176-2e1f-4640-b3fd-637dd506591d\" (UID: \"f10d8176-2e1f-4640-b3fd-637dd506591d\") "
Jul 09 19:28:01.911180 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.911153    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f10d8176-2e1f-4640-b3fd-637dd506591d" (UID: "f10d8176-2e1f-4640-b3fd-637dd506591d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:28:01.911572 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.911543    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f10d8176-2e1f-4640-b3fd-637dd506591d" (UID: "f10d8176-2e1f-4640-b3fd-637dd506591d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:28:01.912968 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.912945    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10d8176-2e1f-4640-b3fd-637dd506591d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f10d8176-2e1f-4640-b3fd-637dd506591d" (UID: "f10d8176-2e1f-4640-b3fd-637dd506591d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:28:01.913073 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:01.912996    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10d8176-2e1f-4640-b3fd-637dd506591d-kube-api-access-gzd8g" (OuterVolumeSpecName: "kube-api-access-gzd8g") pod "f10d8176-2e1f-4640-b3fd-637dd506591d" (UID: "f10d8176-2e1f-4640-b3fd-637dd506591d"). InnerVolumeSpecName "kube-api-access-gzd8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:28:02.012415 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.012334    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10d8176-2e1f-4640-b3fd-637dd506591d-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:02.012415 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.012364    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:02.012415 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.012376    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f10d8176-2e1f-4640-b3fd-637dd506591d-tokenizer-uds\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:02.012415 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.012385    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzd8g\" (UniqueName: \"kubernetes.io/projected/f10d8176-2e1f-4640-b3fd-637dd506591d-kube-api-access-gzd8g\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:02.065035 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.064994    2565 generic.go:356] "Generic (PLEG): container finished" podID="f10d8176-2e1f-4640-b3fd-637dd506591d" containerID="cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4" exitCode=0
Jul 09 19:28:02.065232 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.065050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" event={"ID":"f10d8176-2e1f-4640-b3fd-637dd506591d","Type":"ContainerDied","Data":"cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4"}
Jul 09 19:28:02.065232 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.065064    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"
Jul 09 19:28:02.065232 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.065119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq" event={"ID":"f10d8176-2e1f-4640-b3fd-637dd506591d","Type":"ContainerDied","Data":"66727e11678ddad63cfeac35f320633b15a48949cb3e52ec3cce8f8bec7a3227"}
Jul 09 19:28:02.065232 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.065141    2565 scope.go:117] "RemoveContainer" containerID="cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4"
Jul 09 19:28:02.075765 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.075751    2565 scope.go:117] "RemoveContainer" containerID="9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92"
Jul 09 19:28:02.085109 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.085082    2565 scope.go:117] "RemoveContainer" containerID="cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4"
Jul 09 19:28:02.085512 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:28:02.085474    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4\": container with ID starting with cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4 not found: ID does not exist" containerID="cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4"
Jul 09 19:28:02.085641 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.085514    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4"} err="failed to get container status \"cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4\": rpc error: code = NotFound desc = could not find container \"cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4\": container with ID starting with cff8da58a8744ca903be000a9f6d5b3866a7925b9f8cd8358a58e29a37e154a4 not found: ID does not exist"
Jul 09 19:28:02.085641 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.085540    2565 scope.go:117] "RemoveContainer" containerID="9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92"
Jul 09 19:28:02.085891 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:28:02.085831    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92\": container with ID starting with 9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92 not found: ID does not exist" containerID="9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92"
Jul 09 19:28:02.086356 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.086031    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92"} err="failed to get container status \"9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92\": rpc error: code = NotFound desc = could not find container \"9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92\": container with ID starting with 9b12d84bcd0b62a7c11423c6c93c7fe112f92fb10bf803a2aada0c2289c92e92 not found: ID does not exist"
Jul 09 19:28:02.091508 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.091484    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"]
Jul 09 19:28:02.096367 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.096336    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-59ccfbdd6f-2x6kq"]
Jul 09 19:28:02.868604 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:02.868572    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f10d8176-2e1f-4640-b3fd-637dd506591d" path="/var/lib/kubelet/pods/f10d8176-2e1f-4640-b3fd-637dd506591d/volumes"
Jul 09 19:28:16.510215 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:16.510157    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerName="main" containerID="cri-o://5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01" gracePeriod=45
Jul 09 19:28:18.460803 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.460777    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-5545457c4d-krxfb_2152b310-5619-4ef7-a96d-a3522d3a9afd/main/0.log"
Jul 09 19:28:18.462432 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.462416    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:28:18.559277 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559192    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-home\") pod \"2152b310-5619-4ef7-a96d-a3522d3a9afd\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") "
Jul 09 19:28:18.559277 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559225    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-dshm\") pod \"2152b310-5619-4ef7-a96d-a3522d3a9afd\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") "
Jul 09 19:28:18.559277 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559246    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2152b310-5619-4ef7-a96d-a3522d3a9afd-tls-certs\") pod \"2152b310-5619-4ef7-a96d-a3522d3a9afd\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") "
Jul 09 19:28:18.559277 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559271    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-model-cache\") pod \"2152b310-5619-4ef7-a96d-a3522d3a9afd\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") "
Jul 09 19:28:18.559615 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559289    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-tmp-dir\") pod \"2152b310-5619-4ef7-a96d-a3522d3a9afd\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") "
Jul 09 19:28:18.559615 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559313    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ffpm\" (UniqueName: \"kubernetes.io/projected/2152b310-5619-4ef7-a96d-a3522d3a9afd-kube-api-access-9ffpm\") pod \"2152b310-5619-4ef7-a96d-a3522d3a9afd\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") "
Jul 09 19:28:18.559615 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559336    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-kserve-provision-location\") pod \"2152b310-5619-4ef7-a96d-a3522d3a9afd\" (UID: \"2152b310-5619-4ef7-a96d-a3522d3a9afd\") "
Jul 09 19:28:18.559615 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559516    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-home" (OuterVolumeSpecName: "home") pod "2152b310-5619-4ef7-a96d-a3522d3a9afd" (UID: "2152b310-5619-4ef7-a96d-a3522d3a9afd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:28:18.559615 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559528    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-model-cache" (OuterVolumeSpecName: "model-cache") pod "2152b310-5619-4ef7-a96d-a3522d3a9afd" (UID: "2152b310-5619-4ef7-a96d-a3522d3a9afd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:28:18.559818 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559635    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-home\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:18.559818 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559654    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-model-cache\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:18.559818 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.559737    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2152b310-5619-4ef7-a96d-a3522d3a9afd" (UID: "2152b310-5619-4ef7-a96d-a3522d3a9afd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:28:18.561493 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.561471    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-dshm" (OuterVolumeSpecName: "dshm") pod "2152b310-5619-4ef7-a96d-a3522d3a9afd" (UID: "2152b310-5619-4ef7-a96d-a3522d3a9afd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:28:18.561493 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.561470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2152b310-5619-4ef7-a96d-a3522d3a9afd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2152b310-5619-4ef7-a96d-a3522d3a9afd" (UID: "2152b310-5619-4ef7-a96d-a3522d3a9afd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:28:18.561697 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.561500    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2152b310-5619-4ef7-a96d-a3522d3a9afd-kube-api-access-9ffpm" (OuterVolumeSpecName: "kube-api-access-9ffpm") pod "2152b310-5619-4ef7-a96d-a3522d3a9afd" (UID: "2152b310-5619-4ef7-a96d-a3522d3a9afd"). InnerVolumeSpecName "kube-api-access-9ffpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:28:18.616289 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.616243    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2152b310-5619-4ef7-a96d-a3522d3a9afd" (UID: "2152b310-5619-4ef7-a96d-a3522d3a9afd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:28:18.660960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.660894    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-dshm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:18.660960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.660953    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2152b310-5619-4ef7-a96d-a3522d3a9afd-tls-certs\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:18.661160 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.660986    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-tmp-dir\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:18.661160 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.660995    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ffpm\" (UniqueName: \"kubernetes.io/projected/2152b310-5619-4ef7-a96d-a3522d3a9afd-kube-api-access-9ffpm\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:18.661160 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:18.661004    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2152b310-5619-4ef7-a96d-a3522d3a9afd-kserve-provision-location\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:28:19.142857 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.142780    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-5545457c4d-krxfb_2152b310-5619-4ef7-a96d-a3522d3a9afd/main/0.log"
Jul 09 19:28:19.143183 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.143158    2565 generic.go:356] "Generic (PLEG): container finished" podID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerID="5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01" exitCode=130
Jul 09 19:28:19.143285 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.143216    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"
Jul 09 19:28:19.143329 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.143211    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" event={"ID":"2152b310-5619-4ef7-a96d-a3522d3a9afd","Type":"ContainerDied","Data":"5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01"}
Jul 09 19:28:19.143371 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.143340    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb" event={"ID":"2152b310-5619-4ef7-a96d-a3522d3a9afd","Type":"ContainerDied","Data":"d8ba8ba84b9cc0dce75898c5304a494fcd675e525d944203693b74870bde5642"}
Jul 09 19:28:19.143371 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.143365    2565 scope.go:117] "RemoveContainer" containerID="5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01"
Jul 09 19:28:19.151816 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.151798    2565 scope.go:117] "RemoveContainer" containerID="d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776"
Jul 09 19:28:19.159480 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.159464    2565 scope.go:117] "RemoveContainer" containerID="5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01"
Jul 09 19:28:19.159698 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:28:19.159682    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01\": container with ID starting with 5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01 not found: ID does not exist" containerID="5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01"
Jul 09 19:28:19.159765 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.159705    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01"} err="failed to get container status \"5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01\": rpc error: code = NotFound desc = could not find container \"5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01\": container with ID starting with 5de75c164d14d02264475329fd9e1491a8a800094ff7866102341f5823289c01 not found: ID does not exist"
Jul 09 19:28:19.159765 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.159720    2565 scope.go:117] "RemoveContainer" containerID="d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776"
Jul 09 19:28:19.160012 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:28:19.159993    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776\": container with ID starting with d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776 not found: ID does not exist" containerID="d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776"
Jul 09 19:28:19.160066 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.160021    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776"} err="failed to get container status \"d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776\": rpc error: code = NotFound desc = could not find container \"d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776\": container with ID starting with d989b8a5e88b5c46ea945e4cb4b83430a38e02094ed5fe76b9d06cec3ec75776 not found: ID does not exist"
Jul 09 19:28:19.167452 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.167430    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"]
Jul 09 19:28:19.171521 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:19.171496    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5545457c4d-krxfb"]
Jul 09 19:28:20.868295 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:20.868264    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" path="/var/lib/kubelet/pods/2152b310-5619-4ef7-a96d-a3522d3a9afd/volumes"
Jul 09 19:28:21.024948 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.024914    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ctxwt/must-gather-pv49x"]
Jul 09 19:28:21.025246 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025234    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f10d8176-2e1f-4640-b3fd-637dd506591d" containerName="storage-initializer"
Jul 09 19:28:21.025303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025247    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f10d8176-2e1f-4640-b3fd-637dd506591d" containerName="storage-initializer"
Jul 09 19:28:21.025303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025258    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main"
Jul 09 19:28:21.025303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025264    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main"
Jul 09 19:28:21.025303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025273    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerName="storage-initializer"
Jul 09 19:28:21.025303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025279    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerName="storage-initializer"
Jul 09 19:28:21.025303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025284    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerName="main"
Jul 09 19:28:21.025303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025289    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerName="main"
Jul 09 19:28:21.025303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025303    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f10d8176-2e1f-4640-b3fd-637dd506591d" containerName="main"
Jul 09 19:28:21.025576 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025308    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f10d8176-2e1f-4640-b3fd-637dd506591d" containerName="main"
Jul 09 19:28:21.025576 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025317    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="storage-initializer"
Jul 09 19:28:21.025576 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025321    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="storage-initializer"
Jul 09 19:28:21.025576 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025365    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da0925f-10ca-494b-bb60-79cbb9e0a13e" containerName="main"
Jul 09 19:28:21.025576 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025375    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f10d8176-2e1f-4640-b3fd-637dd506591d" containerName="main"
Jul 09 19:28:21.025576 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.025382    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2152b310-5619-4ef7-a96d-a3522d3a9afd" containerName="main"
Jul 09 19:28:21.028313 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.028297    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:28:21.030816 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.030795    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ctxwt\"/\"openshift-service-ca.crt\""
Jul 09 19:28:21.030933 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.030828    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ctxwt\"/\"kube-root-ca.crt\""
Jul 09 19:28:21.039317 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.039294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ctxwt/must-gather-pv49x"]
Jul 09 19:28:21.080622 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.080588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ssv\" (UniqueName: \"kubernetes.io/projected/72ce733c-ac44-40b8-9753-74461732d971-kube-api-access-r6ssv\") pod \"must-gather-pv49x\" (UID: \"72ce733c-ac44-40b8-9753-74461732d971\") " pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:28:21.080770 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.080660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72ce733c-ac44-40b8-9753-74461732d971-must-gather-output\") pod \"must-gather-pv49x\" (UID: \"72ce733c-ac44-40b8-9753-74461732d971\") " pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:28:21.181734 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.181655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ssv\" (UniqueName: \"kubernetes.io/projected/72ce733c-ac44-40b8-9753-74461732d971-kube-api-access-r6ssv\") pod \"must-gather-pv49x\" (UID: \"72ce733c-ac44-40b8-9753-74461732d971\") " pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:28:21.181734 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.181715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72ce733c-ac44-40b8-9753-74461732d971-must-gather-output\") pod \"must-gather-pv49x\" (UID: \"72ce733c-ac44-40b8-9753-74461732d971\") " pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:28:21.182073 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.182052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72ce733c-ac44-40b8-9753-74461732d971-must-gather-output\") pod \"must-gather-pv49x\" (UID: \"72ce733c-ac44-40b8-9753-74461732d971\") " pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:28:21.190023 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.190002    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ssv\" (UniqueName: \"kubernetes.io/projected/72ce733c-ac44-40b8-9753-74461732d971-kube-api-access-r6ssv\") pod \"must-gather-pv49x\" (UID: \"72ce733c-ac44-40b8-9753-74461732d971\") " pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:28:21.336816 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.336774    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:28:21.469667 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.469637    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ctxwt/must-gather-pv49x"]
Jul 09 19:28:21.471893 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:28:21.471863    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72ce733c_ac44_40b8_9753_74461732d971.slice/crio-4116f0b65a1ada74610ebec179fd7d8dab0aa0928b470486fc6d1cef434d619d WatchSource:0}: Error finding container 4116f0b65a1ada74610ebec179fd7d8dab0aa0928b470486fc6d1cef434d619d: Status 404 returned error can't find the container with id 4116f0b65a1ada74610ebec179fd7d8dab0aa0928b470486fc6d1cef434d619d
Jul 09 19:28:21.473523 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:21.473506    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:28:22.161947 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:22.161848    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ctxwt/must-gather-pv49x" event={"ID":"72ce733c-ac44-40b8-9753-74461732d971","Type":"ContainerStarted","Data":"4116f0b65a1ada74610ebec179fd7d8dab0aa0928b470486fc6d1cef434d619d"}
Jul 09 19:28:28.197281 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:28.197238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ctxwt/must-gather-pv49x" event={"ID":"72ce733c-ac44-40b8-9753-74461732d971","Type":"ContainerStarted","Data":"fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6"}
Jul 09 19:28:28.197281 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:28.197286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ctxwt/must-gather-pv49x" event={"ID":"72ce733c-ac44-40b8-9753-74461732d971","Type":"ContainerStarted","Data":"3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5"}
Jul 09 19:28:28.217410 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:28.217348    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ctxwt/must-gather-pv49x" podStartSLOduration=1.181538806 podStartE2EDuration="7.21732895s" totalImagesPullingTime="6.035790144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:28:28.216119519 +0000 UTC m=+5027.934382285" watchObservedRunningTime="2026-07-09 19:28:28.21732895 +0000 UTC m=+5027.935591714"
Jul 09 19:28:37.071276 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:37.071193    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:37.194924 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:37.194885    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:37.256844 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:37.256811    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:37.257659 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:37.257634    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:38.750127 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:38.750092    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:38.849427 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:38.849397    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:38.897006 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:38.896974    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:38.897303 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:38.897286    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:40.394924 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:40.394889    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:40.477084 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:40.477054    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:40.512328 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:40.512297    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:40.512486 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:40.512453    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:41.988418 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:41.988392    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:42.088698 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:42.088671    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:42.126454 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:42.126431    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:42.126847 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:42.126829    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:43.669346 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:43.669317    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:43.754646 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:43.754610    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:43.790757 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:43.790729    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:43.791033 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:43.791017    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:45.287250 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:45.287224    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:45.372835 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:45.372801    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:45.408831 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:45.408796    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:45.409146 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:45.409108    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:46.892375 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:46.892336    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:46.973787 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:46.973755    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:47.006325 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:47.006286    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:47.006535 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:47.006501    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:48.586962 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:48.586933    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:48.680791 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:48.680758    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:48.717933 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:48.717877    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:48.718099 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:48.718089    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:50.311146 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:50.311117    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:50.454396 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:50.454365    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:50.497889 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:50.495189    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:50.497889 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:50.495628    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:52.014499 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:52.014470    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:52.101182 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:52.101140    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:52.138794 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:52.138768    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:52.139130 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:52.139108    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:53.648772 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:53.648745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:53.737898 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:53.737865    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:53.773200 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:53.773167    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:53.773493 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:53.773476    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:55.264486 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:55.264460    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:55.344411 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:55.344384    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:55.375157 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:55.375128    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:55.375409 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:55.375394    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:56.848275 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:56.848244    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:56.934155 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:56.934129    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:56.970446 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:56.970415    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:56.970734 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:56.970717    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:28:58.446873 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:58.446844    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:28:58.528465 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:58.528436    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:28:58.565727 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:58.565688    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:28:58.566086 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:28:58.566070    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:29:00.065526 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:00.065495    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:29:00.157294 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:00.157257    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:29:00.198370 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:00.198340    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:29:00.198703 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:00.198685    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:29:01.692776 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:01.692751    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-hx52p_d1a50ed2-da02-4097-8351-d11e25d71cb9/storage-initializer/0.log"
Jul 09 19:29:01.772677 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:01.772645    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5_0b12fcf9-7192-4135-a99e-9d03a833fd7b/main/0.log"
Jul 09 19:29:01.805982 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:01.805952    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/1.log"
Jul 09 19:29:01.806241 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:01.806229    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-wl8wv_de7a8044-a4f2-47c6-bbdc-04fec9ff8da4/istio-proxy/0.log"
Jul 09 19:29:08.837035 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:08.836990    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:10.067361 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:10.067313    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:11.287457 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:11.287427    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:12.499022 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:12.498995    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:13.760016 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:13.759983    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:15.078180 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:15.078148    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:16.340291 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:16.340260    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:17.547454 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:17.547421    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:18.774226 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:18.774183    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:19.988960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:19.988932    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:21.204670 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:21.204642    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:22.448353 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:22.448322    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:23.677213 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:23.677177    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:24.892633 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:24.892600    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:26.115853 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:26.115825    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:27.324484 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:27.324457    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:28.542877 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:28.542844    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:29.748752 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:29.748719    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:30.961232 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:30.961206    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:32.176960 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:32.176896    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:33.419742 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:33.419713    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:34.527129 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:34.527042    2565 generic.go:356] "Generic (PLEG): container finished" podID="72ce733c-ac44-40b8-9753-74461732d971" containerID="3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5" exitCode=0
Jul 09 19:29:34.527129 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:34.527116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ctxwt/must-gather-pv49x" event={"ID":"72ce733c-ac44-40b8-9753-74461732d971","Type":"ContainerDied","Data":"3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5"}
Jul 09 19:29:34.527585 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:34.527428    2565 scope.go:117] "RemoveContainer" containerID="3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5"
Jul 09 19:29:35.219457 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:35.219425    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ctxwt_must-gather-pv49x_72ce733c-ac44-40b8-9753-74461732d971/gather/0.log"
Jul 09 19:29:39.066345 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:39.066320    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lznbs_0fe991f9-27b4-4d56-a779-014879464f58/global-pull-secret-syncer/0.log"
Jul 09 19:29:39.226300 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:39.226267    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nh5dl_4991788b-de70-4f49-87f4-cad8cabd7fe3/konnectivity-agent/0.log"
Jul 09 19:29:39.258047 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:39.258019    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-87.ec2.internal_d1ab68799b6023da576fe8586ec62200/haproxy/0.log"
Jul 09 19:29:40.879852 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:40.879802    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-ctxwt/must-gather-pv49x"]
Jul 09 19:29:40.880301 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:40.880001    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-ctxwt/must-gather-pv49x" podUID="72ce733c-ac44-40b8-9753-74461732d971" containerName="copy" containerID="cri-o://fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6" gracePeriod=2
Jul 09 19:29:40.881636 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:40.881613    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-ctxwt/must-gather-pv49x"]
Jul 09 19:29:40.882219 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:29:40.882192    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pv49x\" is forbidden: User \"system:node:ip-10-0-129-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ctxwt\": no relationship found between node 'ip-10-0-129-87.ec2.internal' and this object" podUID="72ce733c-ac44-40b8-9753-74461732d971" pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:29:41.115456 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.115432    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ctxwt_must-gather-pv49x_72ce733c-ac44-40b8-9753-74461732d971/copy/0.log"
Jul 09 19:29:41.119312 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.119287    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ctxwt_must-gather-pv49x_72ce733c-ac44-40b8-9753-74461732d971/copy/0.log"
Jul 09 19:29:41.121512 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.121491    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:29:41.123138 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.123119    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ctxwt_must-gather-pv49x_72ce733c-ac44-40b8-9753-74461732d971/copy/0.log"
Jul 09 19:29:41.123919 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:29:41.123874    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pv49x\" is forbidden: User \"system:node:ip-10-0-129-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ctxwt\": no relationship found between node 'ip-10-0-129-87.ec2.internal' and this object" podUID="72ce733c-ac44-40b8-9753-74461732d971" pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:29:41.132635 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.132579    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:29:41.138727 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.138712    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dlqjl_250e9a05-21d4-45d5-9113-f4594095ffdb/ovn-acl-logging/0.log"
Jul 09 19:29:41.229776 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.229743    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72ce733c-ac44-40b8-9753-74461732d971-must-gather-output\") pod \"72ce733c-ac44-40b8-9753-74461732d971\" (UID: \"72ce733c-ac44-40b8-9753-74461732d971\") "
Jul 09 19:29:41.229998 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.229836    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6ssv\" (UniqueName: \"kubernetes.io/projected/72ce733c-ac44-40b8-9753-74461732d971-kube-api-access-r6ssv\") pod \"72ce733c-ac44-40b8-9753-74461732d971\" (UID: \"72ce733c-ac44-40b8-9753-74461732d971\") "
Jul 09 19:29:41.232048 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.232020    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ce733c-ac44-40b8-9753-74461732d971-kube-api-access-r6ssv" (OuterVolumeSpecName: "kube-api-access-r6ssv") pod "72ce733c-ac44-40b8-9753-74461732d971" (UID: "72ce733c-ac44-40b8-9753-74461732d971"). InnerVolumeSpecName "kube-api-access-r6ssv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:29:41.240838 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.240806    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ce733c-ac44-40b8-9753-74461732d971-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "72ce733c-ac44-40b8-9753-74461732d971" (UID: "72ce733c-ac44-40b8-9753-74461732d971"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:29:41.330444 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.330409    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6ssv\" (UniqueName: \"kubernetes.io/projected/72ce733c-ac44-40b8-9753-74461732d971-kube-api-access-r6ssv\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:29:41.330444 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.330440    2565 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72ce733c-ac44-40b8-9753-74461732d971-must-gather-output\") on node \"ip-10-0-129-87.ec2.internal\" DevicePath \"\""
Jul 09 19:29:41.560560 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.560536    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ctxwt_must-gather-pv49x_72ce733c-ac44-40b8-9753-74461732d971/copy/0.log"
Jul 09 19:29:41.560886 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.560864    2565 generic.go:356] "Generic (PLEG): container finished" podID="72ce733c-ac44-40b8-9753-74461732d971" containerID="fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6" exitCode=143
Jul 09 19:29:41.560958 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.560925    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:29:41.560998 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.560970    2565 scope.go:117] "RemoveContainer" containerID="fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6"
Jul 09 19:29:41.563364 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:29:41.563340    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pv49x\" is forbidden: User \"system:node:ip-10-0-129-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ctxwt\": no relationship found between node 'ip-10-0-129-87.ec2.internal' and this object" podUID="72ce733c-ac44-40b8-9753-74461732d971" pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:29:41.571533 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.571516    2565 scope.go:117] "RemoveContainer" containerID="3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5"
Jul 09 19:29:41.575551 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:29:41.575526    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pv49x\" is forbidden: User \"system:node:ip-10-0-129-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ctxwt\": no relationship found between node 'ip-10-0-129-87.ec2.internal' and this object" podUID="72ce733c-ac44-40b8-9753-74461732d971" pod="openshift-must-gather-ctxwt/must-gather-pv49x"
Jul 09 19:29:41.584710 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.584691    2565 scope.go:117] "RemoveContainer" containerID="fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6"
Jul 09 19:29:41.585131 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:29:41.585110    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6\": container with ID starting with fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6 not found: ID does not exist" containerID="fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6"
Jul 09 19:29:41.585184 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.585141    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6"} err="failed to get container status \"fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6\": rpc error: code = NotFound desc = could not find container \"fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6\": container with ID starting with fac77b6a60f3091a3f60ccd7a0d1746bb4558267fa3e1c7a10b72ba8f54037c6 not found: ID does not exist"
Jul 09 19:29:41.585184 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.585159    2565 scope.go:117] "RemoveContainer" containerID="3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5"
Jul 09 19:29:41.585436 ip-10-0-129-87 kubenswrapper[2565]: E0709 19:29:41.585410    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5\": container with ID starting with 3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5 not found: ID does not exist" containerID="3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5"
Jul 09 19:29:41.585536 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:41.585441    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5"} err="failed to get container status \"3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5\": rpc error: code = NotFound desc = could not find container \"3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5\": container with ID starting with 3f2f2d711bb2f5e43591628b37c412ba0d8ee3ccaf284ded9e50e0ffa2301bc5 not found: ID does not exist"
Jul 09 19:29:42.868669 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:42.868635    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ce733c-ac44-40b8-9753-74461732d971" path="/var/lib/kubelet/pods/72ce733c-ac44-40b8-9753-74461732d971/volumes"
Jul 09 19:29:44.017996 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:44.017966    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-rpjxx_4495f585-5392-43bf-abba-e323b84280e2/manager/0.log"
Jul 09 19:29:45.662118 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:45.662091    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-g7hw9_ec32bc36-b9ed-479c-a795-0f6c735fd0ab/kube-state-metrics/0.log"
Jul 09 19:29:45.683793 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:45.683767    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-g7hw9_ec32bc36-b9ed-479c-a795-0f6c735fd0ab/kube-rbac-proxy-main/0.log"
Jul 09 19:29:45.706996 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:45.706972    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-g7hw9_ec32bc36-b9ed-479c-a795-0f6c735fd0ab/kube-rbac-proxy-self/0.log"
Jul 09 19:29:45.765087 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:45.765054    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-t78pw_f3ab618e-407f-49ca-a5eb-68262997db9e/monitoring-plugin/0.log"
Jul 09 19:29:45.954891 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:45.954858    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2zkd_b5b9872b-eb91-444f-8522-8c0007eb185e/node-exporter/0.log"
Jul 09 19:29:45.980076 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:45.980029    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2zkd_b5b9872b-eb91-444f-8522-8c0007eb185e/kube-rbac-proxy/0.log"
Jul 09 19:29:46.003274 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.003250    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2zkd_b5b9872b-eb91-444f-8522-8c0007eb185e/init-textfile/0.log"
Jul 09 19:29:46.139486 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.139452    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9aabeba0-a5f0-46e8-b6ce-790251d64736/prometheus/0.log"
Jul 09 19:29:46.162685 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.162656    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9aabeba0-a5f0-46e8-b6ce-790251d64736/config-reloader/0.log"
Jul 09 19:29:46.187534 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.187509    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9aabeba0-a5f0-46e8-b6ce-790251d64736/thanos-sidecar/0.log"
Jul 09 19:29:46.216138 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.216067    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9aabeba0-a5f0-46e8-b6ce-790251d64736/kube-rbac-proxy-web/0.log"
Jul 09 19:29:46.241727 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.241701    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9aabeba0-a5f0-46e8-b6ce-790251d64736/kube-rbac-proxy/0.log"
Jul 09 19:29:46.262450 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.262426    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9aabeba0-a5f0-46e8-b6ce-790251d64736/kube-rbac-proxy-thanos/0.log"
Jul 09 19:29:46.290998 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.290974    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9aabeba0-a5f0-46e8-b6ce-790251d64736/init-config-reloader/0.log"
Jul 09 19:29:46.454040 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.454011    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d95b9d954-vjbdq_e24db2ac-e7dd-453b-950b-6ebb00f3952e/telemeter-client/0.log"
Jul 09 19:29:46.486984 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.486833    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d95b9d954-vjbdq_e24db2ac-e7dd-453b-950b-6ebb00f3952e/reload/0.log"
Jul 09 19:29:46.521978 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:46.521954    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d95b9d954-vjbdq_e24db2ac-e7dd-453b-950b-6ebb00f3952e/kube-rbac-proxy/0.log"
Jul 09 19:29:49.190274 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.190244    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-92h5k_edb7714a-acfa-4ee9-b6ad-0579ce172655/download-server/0.log"
Jul 09 19:29:49.424388 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.424351    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"]
Jul 09 19:29:49.424716 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.424677    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ce733c-ac44-40b8-9753-74461732d971" containerName="gather"
Jul 09 19:29:49.424716 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.424688    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ce733c-ac44-40b8-9753-74461732d971" containerName="gather"
Jul 09 19:29:49.424716 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.424707    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ce733c-ac44-40b8-9753-74461732d971" containerName="copy"
Jul 09 19:29:49.424716 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.424712    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ce733c-ac44-40b8-9753-74461732d971" containerName="copy"
Jul 09 19:29:49.424863 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.424765    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ce733c-ac44-40b8-9753-74461732d971" containerName="gather"
Jul 09 19:29:49.424863 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.424775    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ce733c-ac44-40b8-9753-74461732d971" containerName="copy"
Jul 09 19:29:49.431171 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.431146    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.433805 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.433759    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gxtmp\"/\"kube-root-ca.crt\""
Jul 09 19:29:49.435152 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.434955    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gxtmp\"/\"default-dockercfg-vf5cc\""
Jul 09 19:29:49.435152 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.435034    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gxtmp\"/\"openshift-service-ca.crt\""
Jul 09 19:29:49.437620 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.437596    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"]
Jul 09 19:29:49.500356 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.500322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-sys\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.500550 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.500378    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-podres\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.500550 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.500441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcr8h\" (UniqueName: \"kubernetes.io/projected/ceb911a4-b927-463a-b0e1-ba0a062115ac-kube-api-access-bcr8h\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.500550 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.500502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-lib-modules\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.500550 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.500530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-proc\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.601478 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.601447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-sys\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.601651 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.601502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-podres\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.601651 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.601526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcr8h\" (UniqueName: \"kubernetes.io/projected/ceb911a4-b927-463a-b0e1-ba0a062115ac-kube-api-access-bcr8h\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.601651 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.601556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-lib-modules\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.601651 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.601582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-sys\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.601651 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.601591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-proc\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.601853 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.601704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-proc\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.601853 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.601704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-lib-modules\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.601853 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.601824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ceb911a4-b927-463a-b0e1-ba0a062115ac-podres\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.610278 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.610244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcr8h\" (UniqueName: \"kubernetes.io/projected/ceb911a4-b927-463a-b0e1-ba0a062115ac-kube-api-access-bcr8h\") pod \"perf-node-gather-daemonset-5kzwj\" (UID: \"ceb911a4-b927-463a-b0e1-ba0a062115ac\") " pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.742021 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.741985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:49.872343 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:49.872318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"]
Jul 09 19:29:49.874499 ip-10-0-129-87 kubenswrapper[2565]: W0709 19:29:49.874470    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podceb911a4_b927_463a_b0e1_ba0a062115ac.slice/crio-d42fb73a8a611d56cfd03881cb19289c2ac5fa1bf6297152d7e5108bd2c7dbdf WatchSource:0}: Error finding container d42fb73a8a611d56cfd03881cb19289c2ac5fa1bf6297152d7e5108bd2c7dbdf: Status 404 returned error can't find the container with id d42fb73a8a611d56cfd03881cb19289c2ac5fa1bf6297152d7e5108bd2c7dbdf
Jul 09 19:29:50.601321 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:50.601285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj" event={"ID":"ceb911a4-b927-463a-b0e1-ba0a062115ac","Type":"ContainerStarted","Data":"b529f84faf20369a766caee22e0cea4c3db2a0474056da8b18530da32e88f558"}
Jul 09 19:29:50.601835 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:50.601343    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj" event={"ID":"ceb911a4-b927-463a-b0e1-ba0a062115ac","Type":"ContainerStarted","Data":"d42fb73a8a611d56cfd03881cb19289c2ac5fa1bf6297152d7e5108bd2c7dbdf"}
Jul 09 19:29:50.601835 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:50.601415    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:50.617715 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:50.617666    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj" podStartSLOduration=1.617634118 podStartE2EDuration="1.617634118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:29:50.61705883 +0000 UTC m=+5110.335321594" watchObservedRunningTime="2026-07-09 19:29:50.617634118 +0000 UTC m=+5110.335896882"
Jul 09 19:29:50.977348 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:50.977319    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvfdf_3656bece-edae-4e6d-b42a-7adca0774519/dns/0.log"
Jul 09 19:29:51.000376 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:51.000351    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvfdf_3656bece-edae-4e6d-b42a-7adca0774519/kube-rbac-proxy/0.log"
Jul 09 19:29:51.073744 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:51.073719    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4s6pg_3c7b15ac-cf17-4a04-be09-eaadda984155/dns-node-resolver/0.log"
Jul 09 19:29:51.655705 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:51.655668    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55dc77cc58-ckbs6_2e9d83f7-2520-4ba1-ae4b-e924fe978f9d/registry/0.log"
Jul 09 19:29:51.683926 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:51.683886    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2ptrg_364ccd55-cb96-46e5-a706-bcdf005bf715/node-ca/0.log"
Jul 09 19:29:53.814954 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:53.814926    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8l287_e0a920b9-1318-44a0-8026-6f4a0defd42f/serve-healthcheck-canary/0.log"
Jul 09 19:29:54.462450 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:54.462420    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggcq4_53d9f28c-f462-41ba-a0e8-b376d1d7a962/kube-rbac-proxy/0.log"
Jul 09 19:29:54.486718 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:54.486690    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggcq4_53d9f28c-f462-41ba-a0e8-b376d1d7a962/exporter/0.log"
Jul 09 19:29:54.509522 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:54.509479    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggcq4_53d9f28c-f462-41ba-a0e8-b376d1d7a962/extractor/0.log"
Jul 09 19:29:56.615647 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:56.615621    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gxtmp/perf-node-gather-daemonset-5kzwj"
Jul 09 19:29:57.478149 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:57.478116    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9fc7df7df-j8gwx_2f3048f2-55d1-490f-a716-60fa20458deb/manager/0.log"
Jul 09 19:29:58.276422 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:58.276392    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-5t6ql_fe78cc67-8271-4a39-887a-61d35a87e78a/server/0.log"
Jul 09 19:29:58.789601 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:58.789572    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-p2p26_6d00f1c7-bacd-4f5d-a24c-1688ff545add/manager/0.log"
Jul 09 19:29:58.827304 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:58.827279    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-55kmp_9a57269e-81f9-4d91-9e59-05efa447d487/s3-init/0.log"
Jul 09 19:29:58.849491 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:29:58.849469    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-55kmp_9a57269e-81f9-4d91-9e59-05efa447d487/download-hf-model/0.log"
Jul 09 19:30:04.506223 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:04.506191    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z6jvq_1e138bfa-74f8-48c7-baa3-b233cd50bd4a/migrator/0.log"
Jul 09 19:30:04.531465 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:04.531438    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z6jvq_1e138bfa-74f8-48c7-baa3-b233cd50bd4a/graceful-termination/0.log"
Jul 09 19:30:06.052199 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.052168    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4kfcs_506b3c80-5b84-408c-b9ed-55b660477968/kube-multus/0.log"
Jul 09 19:30:06.506151 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.506123    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lq7r_7d764999-f70b-4528-a03e-6e8eae6d6bae/kube-multus-additional-cni-plugins/0.log"
Jul 09 19:30:06.529694 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.529669    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lq7r_7d764999-f70b-4528-a03e-6e8eae6d6bae/egress-router-binary-copy/0.log"
Jul 09 19:30:06.554787 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.554763    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lq7r_7d764999-f70b-4528-a03e-6e8eae6d6bae/cni-plugins/0.log"
Jul 09 19:30:06.577981 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.577952    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lq7r_7d764999-f70b-4528-a03e-6e8eae6d6bae/bond-cni-plugin/0.log"
Jul 09 19:30:06.603869 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.603839    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lq7r_7d764999-f70b-4528-a03e-6e8eae6d6bae/routeoverride-cni/0.log"
Jul 09 19:30:06.625340 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.625314    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lq7r_7d764999-f70b-4528-a03e-6e8eae6d6bae/whereabouts-cni-bincopy/0.log"
Jul 09 19:30:06.648824 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.648797    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lq7r_7d764999-f70b-4528-a03e-6e8eae6d6bae/whereabouts-cni/0.log"
Jul 09 19:30:06.677090 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.677065    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jn9vx_52e1d619-e463-49f5-b06c-f55c9cb8d1f2/network-metrics-daemon/0.log"
Jul 09 19:30:06.697862 ip-10-0-129-87 kubenswrapper[2565]: I0709 19:30:06.697840    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jn9vx_52e1d619-e463-49f5-b06c-f55c9cb8d1f2/kube-rbac-proxy/0.log"