Jul 09 18:03:24.232610 ip-10-0-140-224 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:03:24.641194 ip-10-0-140-224 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:03:24.641194 ip-10-0-140-224 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:03:24.641194 ip-10-0-140-224 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:03:24.641194 ip-10-0-140-224 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:03:24.641194 ip-10-0-140-224 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:03:24.643773 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.643688    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:03:24.648509 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648494    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:24.648509 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648510    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648514    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648517    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648520    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648522    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648525    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648528    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648531    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648534    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648536    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648539    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648542    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648545    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648547    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648550    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648554    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648558    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648562    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:24.648567 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648565    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648582    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648586    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648589    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648592    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648594    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648598    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648600    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648603    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648606    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648608    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648611    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648614    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648617    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648621    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648625    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648628    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648631    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648633    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648636    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:24.649082 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648638    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648641    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648643    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648645    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648650    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648654    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648657    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648661    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648664    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648667    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648669    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648671    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648674    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648676    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648680    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648682    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648685    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648688    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648690    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648693    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:24.649823 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648695    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648698    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648701    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648703    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648706    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648710    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648713    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648716    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648718    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648721    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648723    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648726    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648728    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648731    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648733    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648736    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648739    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648741    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648744    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:24.650347 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648746    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648749    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648752    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648754    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648757    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648759    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648762    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648768    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648770    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648773    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648775    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648777    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648780    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648782    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648786    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648788    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648791    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648793    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648796    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648798    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:24.651103 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.648801    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651653    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651670    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651676    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651681    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651686    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651691    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651695    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651700    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651704    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651708    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651711    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651715    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651719    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651723    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651727    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651731    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651734    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651738    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651742    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:24.652086 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651745    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651750    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651754    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651757    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651761    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651767    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651779    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651784    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651788    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651792    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651796    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651800    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651805    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651810    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651814    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651819    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651823    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651827    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:24.653001 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651831    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651838    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651843    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651848    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651853    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651857    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651861    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651866    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651870    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651874    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651878    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651882    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651886    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651890    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651894    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651898    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651902    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651909    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651914    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:24.653835 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651919    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651924    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651928    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651933    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651937    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651941    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651946    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651950    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651955    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651960    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651964    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651969    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651973    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651977    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651981    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651986    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.651990    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652000    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652005    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652009    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:24.654673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652013    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652017    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652022    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652026    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652030    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652034    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652038    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652043    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652050    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652055    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652059    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652064    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652068    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652072    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652076    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652080    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652084    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652088    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652092    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652097    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:24.655385 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652101    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652105    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652110    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652777    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652788    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652792    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652797    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652802    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652806    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652810    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652818    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652824    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652829    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652834    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652839    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652843    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652848    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652852    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652857    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652862    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:24.655884 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652866    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652871    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652875    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652879    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652883    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652887    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652892    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652896    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652900    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652904    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652909    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652914    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652921    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652926    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652932    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652936    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652941    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652945    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652949    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652954    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:24.656487 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652958    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652963    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652967    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652972    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652976    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652980    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652985    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652989    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652993    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.652997    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653001    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653005    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653010    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653014    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653018    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653022    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653026    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653030    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653036    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653040    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:24.657186 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653045    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653049    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653053    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653058    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653063    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653067    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653071    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653077    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653081    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653085    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653090    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653094    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653098    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653102    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653107    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653111    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653115    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653119    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653123    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653127    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:24.657706 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653132    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653136    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653140    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653144    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653148    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653153    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653157    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653161    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653165    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653169    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653173    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653178    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653182    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653186    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653190    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653194    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653200    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653204    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653208    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:24.658237 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653212    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653216    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.653220    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653594    2558 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653610    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653623    2558 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653630    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653637    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653642    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653649    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653655    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653660    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653665    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653671    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653677    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653682    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653687    2558 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653691    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653696    2558 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653703    2558 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653708    2558 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653714    2558 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653719    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653724    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:03:24.658761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653729    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653734    2558 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653739    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653756    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653764    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653769    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653774    2558 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653778    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653783    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653788    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653792    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653799    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653804    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653809    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653813    2558 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653818    2558 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653822    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653830    2558 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653834    2558 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653839    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653843    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653848    2558 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653854    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653858    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653863    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:03:24.659352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653868    2558 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653872    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653877    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653882    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653887    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653892    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653896    2558 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653901    2558 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653906    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653911    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653916    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653925    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653930    2558 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653935    2558 flags.go:64] FLAG: --help="false"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653939    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-140-224.ec2.internal"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653944    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653949    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653953    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653959    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653964    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653968    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653973    2558 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653978    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:03:24.660004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653982    2558 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653987    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653991    2558 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.653996    2558 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654001    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654005    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654010    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654015    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654019    2558 flags.go:64] FLAG: --lock-file=""
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654024    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654028    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654033    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654042    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654047    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654052    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654057    2558 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654061    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654067    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654071    2558 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654076    2558 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654084    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654089    2558 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654095    2558 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654100    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654105    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:03:24.660624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654110    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654114    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654119    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654123    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654128    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654141    2558 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654146    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654151    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654155    2558 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654160    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654168    2558 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654173    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654178    2558 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654183    2558 flags.go:64] FLAG: --port="10250"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654188    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654193    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09af8eb96b89b840f"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654198    2558 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654203    2558 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654207    2558 flags.go:64] FLAG: --register-node="true"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654212    2558 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654224    2558 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654230    2558 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654235    2558 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654239    2558 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654245    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:03:24.661253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654250    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654255    2558 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654260    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654266    2558 flags.go:64] FLAG: --runonce="false"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654270    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654275    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654280    2558 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654284    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654289    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654294    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654299    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654304    2558 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654309    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654313    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654318    2558 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654342    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654348    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654353    2558 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654357    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654369    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654373    2558 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654378    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654385    2558 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654389    2558 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654393    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:03:24.661998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654398    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654403    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654408    2558 flags.go:64] FLAG: --v="2"
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654414    2558 flags.go:64] FLAG: --version="false"
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654422    2558 flags.go:64] FLAG: --vmodule=""
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654428    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.654433    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654630    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654638    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654643    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654651    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654657    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654662    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654666    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654671    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654676    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654680    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654686    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654691    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654696    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654700    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:24.662636 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654705    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654710    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654714    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654719    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654723    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654730    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654734    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654738    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654742    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654746    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654750    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654757    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654762    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654767    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654772    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654778    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654783    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654788    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654792    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:24.663143 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654796    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654801    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654807    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654812    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654816    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654821    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654825    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654829    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654833    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654837    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654841    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654847    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654853    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654858    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654862    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654867    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654871    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654875    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654881    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654886    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:24.663662 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654890    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654894    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654898    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654902    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654906    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654910    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654914    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654918    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654923    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654927    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654932    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654936    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654940    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654944    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654949    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654954    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654958    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654962    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654966    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654970    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:24.664153 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654974    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654978    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654982    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654986    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654990    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654994    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.654998    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655002    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655006    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655010    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655016    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655021    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655024    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655028    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655032    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655036    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655040    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655045    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655048    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655053    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:24.664671 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655057    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655061    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655065    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655073    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655077    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.655081    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.655867    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.662862    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.662877    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.663767    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.663774    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663832    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663838    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663842    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:24.665163 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663846    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663849    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663851    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663855    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663857    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663860    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663863    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663865    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663868    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663870    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663873    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663875    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663878    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663881    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663883    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663885    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663888    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663891    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663893    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:24.665556 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663896    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663898    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663901    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663904    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663906    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663908    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663911    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663914    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663917    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663920    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663922    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663924    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663927    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663929    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663932    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663934    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663936    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663939    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663943    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663946    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:24.666028 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663949    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663952    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663955    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663957    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663959    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663962    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663965    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663969    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663972    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663974    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663977    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663979    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663982    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663984    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663987    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663990    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663993    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663995    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.663998    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664001    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:24.666532 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664003    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664006    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664008    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664011    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664013    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664016    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664019    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664021    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664023    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664026    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664028    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664031    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664033    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664036    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664038    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664041    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664043    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664046    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664048    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664050    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:24.667031 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664053    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664055    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664057    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664060    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664062    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664065    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664068    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664071    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664074    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664076    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664079    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664081    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664084    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664086    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664088    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664091    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:24.667524 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664093    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.664098    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664207    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664211    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664214    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664217    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664219    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664222    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664225    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664228    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664231    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664233    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664236    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664239    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664241    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:24.667923 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664244    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664247    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664250    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664254    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664256    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664259    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664262    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664264    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664267    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664270    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664273    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664276    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664278    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664280    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664283    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664285    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664288    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664290    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664292    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664295    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:24.668292 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664297    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664300    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664302    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664304    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664307    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664310    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664312    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664315    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664317    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664319    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664337    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664342    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664345    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664349    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664352    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664356    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664359    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664362    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664365    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:24.668819 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664368    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664371    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664374    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664377    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664380    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664383    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664386    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664388    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664391    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664393    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664396    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664398    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664401    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664403    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664405    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664408    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664410    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664413    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664415    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:24.669274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664418    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664420    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664423    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664425    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664428    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664430    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664433    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664435    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664438    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664440    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664442    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664445    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664447    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664450    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664452    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664455    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664458    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664461    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664463    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664466    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:24.669846 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664468    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664471    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664473    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664475    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664478    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664480    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664483    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:24.664485    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.664491    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.664595    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.666765    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.668394    2558 server.go:1019] "Starting client certificate rotation"
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.668492    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:03:24.670344 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.668521    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:03:24.671257 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.671244    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:03:24.672406 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.672372    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:03:24.684944 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.684919    2558 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:03:24.687378 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.687362    2558 log.go:25] "Validated CRI v1 image API"
Jul 09 18:03:24.688244 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.688228    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:03:24.692358 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.692318    2558 fs.go:135] Filesystem UUIDs: map[34830d7a-ab1d-4efd-967d-38af333e8424:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cad59f35-184a-43e6-8f49-08c48cd7f93a:/dev/nvme0n1p4]
Jul 09 18:03:24.692440 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.692358    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:03:24.698037 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.697922    2558 manager.go:216] Machine: {Timestamp:2026-07-09 18:03:24.695962313 +0000 UTC m=+0.341955843 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098964 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec286ecbfe7eed6eeb40514e25fe62e3 SystemUUID:ec286ecb-fe7e-ed6e-eb40-514e25fe62e3 BootID:66843223-6d14-4277-b54a-8bce26fb2414 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:af:67:5d:7c:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:af:67:5d:7c:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:75:78:63:ce:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:03:24.698037 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.698027    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:03:24.698199 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.698126    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:03:24.699135 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.699116    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:03:24.701030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.701004    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:03:24.701199 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.701036    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-224.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:03:24.701956 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.701944    2558 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:03:24.702017 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.701962    2558 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:03:24.702017 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.701971    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:03:24.703966 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.703954    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:03:24.705317 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.705305    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:03:24.708119 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.708105    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:03:24.708249 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.708238    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:03:24.708304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.708277    2558 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:03:24.708304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.708298    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:03:24.708416 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.708318    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:03:24.708416 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.708350    2558 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:03:24.708416 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.708363    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:03:24.708720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.708706    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:03:24.708788 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.708731    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:03:24.715616 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.715601    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:03:24.717641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.717626    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:03:24.717707 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.717672    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:03:24.719803 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719773    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:03:24.719874 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719811    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:03:24.719874 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719821    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:03:24.719874 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719831    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:03:24.719874 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719843    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:03:24.719874 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719858    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:03:24.719874 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719870    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:03:24.719874 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719878    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:03:24.720157 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719903    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:03:24.720157 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719911    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:03:24.720157 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719940    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:03:24.720157 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.719948    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:03:24.720157 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.720038    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:03:24.720157 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.720046    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:03:24.720631 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.720598    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rgm4m"
Jul 09 18:03:24.722731 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.722714    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-224.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:03:24.723479 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.723456    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:03:24.723479 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.723468    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:03:24.723685 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.723671    2558 server.go:1268] "Started kubelet"
Jul 09 18:03:24.723808 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.723755    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:03:24.723896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.723826    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:03:24.723948 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.723757    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:03:24.724559 ip-10-0-140-224 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:03:24.725089 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.725072    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:03:24.726045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.726029    2558 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:03:24.727920 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.727904    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rgm4m"
Jul 09 18:03:24.730155 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.730135    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:03:24.732788 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.732758    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:03:24.732788 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.732765    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:03:24.732924 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.732870    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:24.734785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.733490    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:03:24.734785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.733507    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:03:24.734785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.733632    2558 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:03:24.734785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.733640    2558 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:03:24.735182 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.734789    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:03:24.735182 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.735028    2558 factory.go:153] Registering CRI-O factory
Jul 09 18:03:24.735182 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.735044    2558 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:03:24.735182 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.735119    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:03:24.735182 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.735131    2558 factory.go:55] Registering systemd factory
Jul 09 18:03:24.735182 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.735142    2558 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:03:24.735182 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.735164    2558 factory.go:103] Registering Raw factory
Jul 09 18:03:24.735596 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.735194    2558 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:03:24.736064 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.736046    2558 manager.go:349] Starting recovery of all containers
Jul 09 18:03:24.738722 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.738702    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:24.744962 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.744795    2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-224.ec2.internal\" not found" node="ip-10-0-140-224.ec2.internal"
Jul 09 18:03:24.748662 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.748646    2558 manager.go:354] Recovery completed
Jul 09 18:03:24.752596 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.752580    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:24.754470 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.754454    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:24.754561 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.754485    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:24.754561 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.754500    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:24.755048 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.755030    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:03:24.755048 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.755046    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:24.755208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.755046    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:03:24.755208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.755121    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:03:24.755778 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.755759    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:24.755861 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.755793    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:24.755861 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.755806    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:24.756354 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.756342    2558 policy_none.go:49] "None policy: Start"
Jul 09 18:03:24.756407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.756358    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:03:24.756407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.756369    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:03:24.757339 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.757314    2558 policy_none.go:47] "Start"
Jul 09 18:03:24.794529 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.794338    2558 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:03:24.794529 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.794391    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:03:24.794529 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.794405    2558 server.go:87] "Starting device plugin registration server"
Jul 09 18:03:24.811175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.794691    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:03:24.811175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.794711    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:03:24.811175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.795474    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:03:24.811175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.795573    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:03:24.811175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.795584    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:03:24.811175 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.795633    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:03:24.811175 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.795684    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:24.885572 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.885530    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:03:24.886956 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.886930    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:03:24.887074 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.886963    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:03:24.887074 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.886984    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:03:24.887074 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.887019    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:03:24.889696 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.889671    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:24.894841 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.894781    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:24.895843 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.895828    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:24.895923 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.895857    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:24.895923 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.895869    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:24.895923 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.895899    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-224.ec2.internal"
Jul 09 18:03:24.904824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.904808    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-224.ec2.internal"
Jul 09 18:03:24.904874 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.904833    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-224.ec2.internal\": node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:24.921112 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:24.921088    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:24.987313 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.987270    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal"]
Jul 09 18:03:24.987424 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.987383    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:24.988389 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.988369    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:24.988509 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.988404    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:24.988509 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.988419    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:24.989905 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.989891    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:24.990064 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.990051    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:24.990099 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.990079    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:24.990616 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.990600    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:24.990715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.990603    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:24.990715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.990631    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:24.990715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.990645    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:24.990715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.990650    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:24.990715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.990659    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:24.991891 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.991874    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:24.991962 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.991903    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:24.994254 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.994116    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:24.994640 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.994628    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:24.994832 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:24.994649    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:25.005859 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.005842    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-224.ec2.internal\" not found" node="ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.011052 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.011037    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-224.ec2.internal\" not found" node="ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.021259 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.021244    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:25.121535 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.121508    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:25.135666 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.135642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d2fa477e951a5bf596a004d58a807d25-config\") pod \"kube-apiserver-proxy-ip-10-0-140-224.ec2.internal\" (UID: \"d2fa477e951a5bf596a004d58a807d25\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.135796 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.135673    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac32b18b87f72dcc3ed24191f7feefea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal\" (UID: \"ac32b18b87f72dcc3ed24191f7feefea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.135796 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.135695    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac32b18b87f72dcc3ed24191f7feefea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal\" (UID: \"ac32b18b87f72dcc3ed24191f7feefea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.222636 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.222560    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:25.235920 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.235891    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d2fa477e951a5bf596a004d58a807d25-config\") pod \"kube-apiserver-proxy-ip-10-0-140-224.ec2.internal\" (UID: \"d2fa477e951a5bf596a004d58a807d25\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.236006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.235923    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac32b18b87f72dcc3ed24191f7feefea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal\" (UID: \"ac32b18b87f72dcc3ed24191f7feefea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.236006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.235943    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac32b18b87f72dcc3ed24191f7feefea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal\" (UID: \"ac32b18b87f72dcc3ed24191f7feefea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.236006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.235989    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac32b18b87f72dcc3ed24191f7feefea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal\" (UID: \"ac32b18b87f72dcc3ed24191f7feefea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.236006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.235994    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ac32b18b87f72dcc3ed24191f7feefea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal\" (UID: \"ac32b18b87f72dcc3ed24191f7feefea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.236146 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.235999    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d2fa477e951a5bf596a004d58a807d25-config\") pod \"kube-apiserver-proxy-ip-10-0-140-224.ec2.internal\" (UID: \"d2fa477e951a5bf596a004d58a807d25\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.308100 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.308057    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.314061 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.314040    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.323067 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.323047    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:25.423635 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.423598    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:25.524207 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.524144    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:25.624886 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.624862    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-224.ec2.internal\" not found"
Jul 09 18:03:25.667372 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.667356    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:03:25.667960 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.667483    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:03:25.667960 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.667509    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:03:25.696848 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.696825    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:25.703697 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.703675    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:25.708625 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.708609    2558 apiserver.go:52] "Watching apiserver"
Jul 09 18:03:25.719420 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.719401    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:03:25.720568 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.720549    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-cbvg5","openshift-multus/network-metrics-daemon-5kshf","openshift-network-operator/iptables-alerter-hh2wt","kube-system/konnectivity-agent-dgwzk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c","openshift-network-diagnostics/network-check-target-tsz48","openshift-ovn-kubernetes/ovnkube-node-5s8lz","kube-system/global-pull-secret-syncer-ql4td","openshift-cluster-node-tuning-operator/tuned-nvswk","openshift-dns/node-resolver-qpt8z","openshift-image-registry/node-ca-ft4xk","openshift-multus/multus-54sxh"]
Jul 09 18:03:25.722094 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.722081    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.723258 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.723238    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:25.723363 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.723320    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:25.724572 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.724558    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.725494 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.725478    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:03:25.725782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.725765    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:03:25.725865 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.725783    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:03:25.725969 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.725909    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:25.725969 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.725950    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:03:25.726093 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.726078    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mc6dr\""
Jul 09 18:03:25.726189 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.726171    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:03:25.727126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.727107    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.728003 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.727986    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:25.728354 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.728342    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:25.728415 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.728401    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:25.729607 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.729587    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.729892 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.729827    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:03:25.729979 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.729929    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:25.730040 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.729899    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:03:25.730111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.730098    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vt9gw\""
Jul 09 18:03:25.730202 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.730190    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7n5kq\""
Jul 09 18:03:25.730411 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.730396    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:03:25.730911 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.730896    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:25.730986 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.730953    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:25.731377 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.731297    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:03:25.731377 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.731280    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:58:25 +0000 UTC" deadline="2027-12-04 04:01:45.211552849 +0000 UTC"
Jul 09 18:03:25.731377 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.731304    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:03:25.731377 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.731340    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12297h58m19.480234677s"
Jul 09 18:03:25.731377 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.731363    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:03:25.731377 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.731375    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9m72l\""
Jul 09 18:03:25.732356 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.732313    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.732800 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.732784    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:03:25.732889 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.732805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:03:25.732942 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.732910    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:03:25.732942 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.732932    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.733874 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.733859    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.734231 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.734211    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j7xkq\""
Jul 09 18:03:25.734304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.734242    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:03:25.734304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.734259    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:03:25.734304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.734211    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:03:25.734553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.734538    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:03:25.735011 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.734997    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.736608 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.736388    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:25.736608 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.736449    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:25.737610 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.736932    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dxt28\""
Jul 09 18:03:25.737610 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.737093    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:03:25.737610 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.737225    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:03:25.737741 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.737670    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-skzfs\""
Jul 09 18:03:25.738550 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738045    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.738646 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/283ad84b-60d3-43ab-80e6-33da3e63bc77-kubelet-config\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:25.738712 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738639    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-os-release\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.738712 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738680    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.738813 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738766    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b892b08-6ba5-42bc-84b6-85f34f7d8119-ovnkube-config\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.738813 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738804    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:03:25.738907 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh4hg\" (UniqueName: \"kubernetes.io/projected/c7fde25b-60ca-405a-acf7-53e67418af9f-kube-api-access-zh4hg\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:25.738907 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738878    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:03:25.739006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738804    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:03:25.739006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738954    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-device-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.739006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.738993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-sys-fs\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.739143 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739023    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-run-netns\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.739143 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-cni-bin\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.739143 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g669d\" (UniqueName: \"kubernetes.io/projected/cd1e2564-98b3-4035-819c-830ebcc71c13-kube-api-access-g669d\") pod \"node-ca-ft4xk\" (UID: \"cd1e2564-98b3-4035-819c-830ebcc71c13\") " pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.739269 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739169    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-node-log\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.739269 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739211    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/968af6c7-fb8f-4937-a490-2f31ecdae078-tmp-dir\") pod \"node-resolver-qpt8z\" (UID: \"968af6c7-fb8f-4937-a490-2f31ecdae078\") " pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.739269 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-system-cni-dir\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.739462 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739279    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-cnibin\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.739462 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739346    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w274c\" (UniqueName: \"kubernetes.io/projected/a677a1da-0454-4e1e-9842-729623211e74-kube-api-access-w274c\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.739732 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-kubelet\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.739785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739756    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-registration-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.739785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739780    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-sysctl-conf\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.739881 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739801    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-systemd\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.739881 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739830    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-lib-modules\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.739881 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739857    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740013 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739883    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.740013 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a35e3606-7da3-4b0a-8c77-ebcef77fc51e-konnectivity-ca\") pod \"konnectivity-agent-dgwzk\" (UID: \"a35e3606-7da3-4b0a-8c77-ebcef77fc51e\") " pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:25.740013 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739927    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-log-socket\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740013 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739949    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g96x9\" (UniqueName: \"kubernetes.io/projected/6b892b08-6ba5-42bc-84b6-85f34f7d8119-kube-api-access-g96x9\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740013 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.739991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-host\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b892b08-6ba5-42bc-84b6-85f34f7d8119-ovn-node-metrics-cert\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740057    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a677a1da-0454-4e1e-9842-729623211e74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740078    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-run\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740099    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/559e8c07-add7-41ff-848b-fa5ed81dc791-iptables-alerter-script\") pod \"iptables-alerter-hh2wt\" (UID: \"559e8c07-add7-41ff-848b-fa5ed81dc791\") " pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740108    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w47d9\""
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-run-ovn\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-kubernetes\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740197    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-sys\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.740225 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a35e3606-7da3-4b0a-8c77-ebcef77fc51e-agent-certs\") pod \"konnectivity-agent-dgwzk\" (UID: \"a35e3606-7da3-4b0a-8c77-ebcef77fc51e\") " pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740243    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-run-systemd\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b892b08-6ba5-42bc-84b6-85f34f7d8119-ovnkube-script-lib\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4vc\" (UniqueName: \"kubernetes.io/projected/559e8c07-add7-41ff-848b-fa5ed81dc791-kube-api-access-6j4vc\") pod \"iptables-alerter-hh2wt\" (UID: \"559e8c07-add7-41ff-848b-fa5ed81dc791\") " pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740296    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-systemd-units\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-var-lib-openvswitch\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/283ad84b-60d3-43ab-80e6-33da3e63bc77-dbus\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqm2\" (UniqueName: \"kubernetes.io/projected/bf3fb400-bb4c-49a8-96ba-a046356ba179-kube-api-access-fbqm2\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd1e2564-98b3-4035-819c-830ebcc71c13-host\") pod \"node-ca-ft4xk\" (UID: \"cd1e2564-98b3-4035-819c-830ebcc71c13\") " pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/559e8c07-add7-41ff-848b-fa5ed81dc791-host-slash\") pod \"iptables-alerter-hh2wt\" (UID: \"559e8c07-add7-41ff-848b-fa5ed81dc791\") " pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740464    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-run-openvswitch\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-cni-netd\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/968af6c7-fb8f-4937-a490-2f31ecdae078-hosts-file\") pod \"node-resolver-qpt8z\" (UID: \"968af6c7-fb8f-4937-a490-2f31ecdae078\") " pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a677a1da-0454-4e1e-9842-729623211e74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.740587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740613    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-modprobe-d\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm46g\" (UniqueName: \"kubernetes.io/projected/6c0994d4-a4f2-4581-bdf3-e188ef354645-kube-api-access-bm46g\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740659    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b892b08-6ba5-42bc-84b6-85f34f7d8119-env-overrides\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740683    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740705    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd1e2564-98b3-4035-819c-830ebcc71c13-serviceca\") pod \"node-ca-ft4xk\" (UID: \"cd1e2564-98b3-4035-819c-830ebcc71c13\") " pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740727    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-slash\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrn7\" (UniqueName: \"kubernetes.io/projected/968af6c7-fb8f-4937-a490-2f31ecdae078-kube-api-access-lqrn7\") pod \"node-resolver-qpt8z\" (UID: \"968af6c7-fb8f-4937-a490-2f31ecdae078\") " pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740794    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a677a1da-0454-4e1e-9842-729623211e74-cni-binary-copy\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740825    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-tuned\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740848    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-sysconfig\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740864    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-sysctl-d\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740887    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-run-ovn-kubernetes\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740907    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-socket-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740929    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-var-lib-kubelet\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740955    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c0994d4-a4f2-4581-bdf3-e188ef354645-tmp\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740971    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:03:25.741141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.740998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:25.741642 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.741033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-etc-openvswitch\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.741776 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.741762    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4kzsw\""
Jul 09 18:03:25.746867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.746849    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal"]
Jul 09 18:03:25.748167 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.748156    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:03:25.748278 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.748265    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal"
Jul 09 18:03:25.749420 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.749402    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:03:25.758216 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.758198    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:03:25.758302 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.758241    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal"]
Jul 09 18:03:25.772140 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.772123    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2nr8k"
Jul 09 18:03:25.780965 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.780944    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2nr8k"
Jul 09 18:03:25.835393 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.835372    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:03:25.841768 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.841740    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-daemon-config\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.841892 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.841779    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-etc-openvswitch\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.841892 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.841847    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/283ad84b-60d3-43ab-80e6-33da3e63bc77-kubelet-config\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:25.841892 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.841873    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-etc-openvswitch\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.842045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.841914    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-os-release\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.842045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.841942    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/283ad84b-60d3-43ab-80e6-33da3e63bc77-kubelet-config\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:25.842045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.841970    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.842045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.841999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-cnibin\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.842045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842021    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-socket-dir-parent\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.842045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842034    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-hostroot\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.842322 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-conf-dir\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.842322 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842058    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-os-release\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.842322 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b892b08-6ba5-42bc-84b6-85f34f7d8119-ovnkube-config\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.842322 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.842322 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh4hg\" (UniqueName: \"kubernetes.io/projected/c7fde25b-60ca-405a-acf7-53e67418af9f-kube-api-access-zh4hg\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:25.842322 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-device-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.842322 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-sys-fs\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.842322 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842297    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-device-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842342    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-etc-kubernetes\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-run-netns\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842392    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-cni-bin\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842395    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-sys-fs\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842406    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g669d\" (UniqueName: \"kubernetes.io/projected/cd1e2564-98b3-4035-819c-830ebcc71c13-kube-api-access-g669d\") pod \"node-ca-ft4xk\" (UID: \"cd1e2564-98b3-4035-819c-830ebcc71c13\") " pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842430    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-run-netns\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842434    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-run-netns\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-var-lib-cni-multus\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842451    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-cni-bin\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842538    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-node-log\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842569    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/968af6c7-fb8f-4937-a490-2f31ecdae078-tmp-dir\") pod \"node-resolver-qpt8z\" (UID: \"968af6c7-fb8f-4937-a490-2f31ecdae078\") " pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842570    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-node-log\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-system-cni-dir\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842683    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-cnibin\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842700    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w274c\" (UniqueName: \"kubernetes.io/projected/a677a1da-0454-4e1e-9842-729623211e74-kube-api-access-w274c\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.842698 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842703    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-system-cni-dir\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842709    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b892b08-6ba5-42bc-84b6-85f34f7d8119-ovnkube-config\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842739    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-cnibin\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842766    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-cni-dir\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842785    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxhf5\" (UniqueName: \"kubernetes.io/projected/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-kube-api-access-gxhf5\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-kubelet\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-registration-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842850    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-sysctl-conf\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842866    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-systemd\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-kubelet\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842884    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/968af6c7-fb8f-4937-a490-2f31ecdae078-tmp-dir\") pod \"node-resolver-qpt8z\" (UID: \"968af6c7-fb8f-4937-a490-2f31ecdae078\") " pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842915    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-lib-modules\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842921    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-systemd\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-registration-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-system-cni-dir\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842983    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-sysctl-conf\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.843279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.842986    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-lib-modules\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843019    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a35e3606-7da3-4b0a-8c77-ebcef77fc51e-konnectivity-ca\") pod \"konnectivity-agent-dgwzk\" (UID: \"a35e3606-7da3-4b0a-8c77-ebcef77fc51e\") " pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843043    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843087    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-log-socket\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843114    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g96x9\" (UniqueName: \"kubernetes.io/projected/6b892b08-6ba5-42bc-84b6-85f34f7d8119-kube-api-access-g96x9\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843122    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-log-socket\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843151    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-host\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b892b08-6ba5-42bc-84b6-85f34f7d8119-ovn-node-metrics-cert\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a677a1da-0454-4e1e-9842-729623211e74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-run\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843244    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-host\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/559e8c07-add7-41ff-848b-fa5ed81dc791-iptables-alerter-script\") pod \"iptables-alerter-hh2wt\" (UID: \"559e8c07-add7-41ff-848b-fa5ed81dc791\") " pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-run\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843418    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-run-ovn\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a677a1da-0454-4e1e-9842-729623211e74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843482    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-kubernetes\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-sys\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-run-ovn\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-kubernetes\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-sys\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843676    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a35e3606-7da3-4b0a-8c77-ebcef77fc51e-agent-certs\") pod \"konnectivity-agent-dgwzk\" (UID: \"a35e3606-7da3-4b0a-8c77-ebcef77fc51e\") " pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843738    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-var-lib-cni-bin\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843754    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-run-multus-certs\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843768    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a35e3606-7da3-4b0a-8c77-ebcef77fc51e-konnectivity-ca\") pod \"konnectivity-agent-dgwzk\" (UID: \"a35e3606-7da3-4b0a-8c77-ebcef77fc51e\") " pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-run-systemd\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b892b08-6ba5-42bc-84b6-85f34f7d8119-ovnkube-script-lib\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843838    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a677a1da-0454-4e1e-9842-729623211e74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4vc\" (UniqueName: \"kubernetes.io/projected/559e8c07-add7-41ff-848b-fa5ed81dc791-kube-api-access-6j4vc\") pod \"iptables-alerter-hh2wt\" (UID: \"559e8c07-add7-41ff-848b-fa5ed81dc791\") " pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843892    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-os-release\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843918    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-run-k8s-cni-cncf-io\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-systemd-units\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843981    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-var-lib-openvswitch\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.844964 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.843999    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844014    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/559e8c07-add7-41ff-848b-fa5ed81dc791-iptables-alerter-script\") pod \"iptables-alerter-hh2wt\" (UID: \"559e8c07-add7-41ff-848b-fa5ed81dc791\") " pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844085    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-systemd-units\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844103    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-run-systemd\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844120    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-var-lib-openvswitch\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/283ad84b-60d3-43ab-80e6-33da3e63bc77-dbus\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844178    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqm2\" (UniqueName: \"kubernetes.io/projected/bf3fb400-bb4c-49a8-96ba-a046356ba179-kube-api-access-fbqm2\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844249    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd1e2564-98b3-4035-819c-830ebcc71c13-host\") pod \"node-ca-ft4xk\" (UID: \"cd1e2564-98b3-4035-819c-830ebcc71c13\") " pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/283ad84b-60d3-43ab-80e6-33da3e63bc77-dbus\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/559e8c07-add7-41ff-848b-fa5ed81dc791-host-slash\") pod \"iptables-alerter-hh2wt\" (UID: \"559e8c07-add7-41ff-848b-fa5ed81dc791\") " pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/559e8c07-add7-41ff-848b-fa5ed81dc791-host-slash\") pod \"iptables-alerter-hh2wt\" (UID: \"559e8c07-add7-41ff-848b-fa5ed81dc791\") " pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844360    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd1e2564-98b3-4035-819c-830ebcc71c13-host\") pod \"node-ca-ft4xk\" (UID: \"cd1e2564-98b3-4035-819c-830ebcc71c13\") " pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-run-openvswitch\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-cni-netd\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/968af6c7-fb8f-4937-a490-2f31ecdae078-hosts-file\") pod \"node-resolver-qpt8z\" (UID: \"968af6c7-fb8f-4937-a490-2f31ecdae078\") " pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844521    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-run-openvswitch\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844530    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b892b08-6ba5-42bc-84b6-85f34f7d8119-ovnkube-script-lib\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.845853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844542    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a677a1da-0454-4e1e-9842-729623211e74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844549    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-cni-netd\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844593    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-modprobe-d\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844656    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm46g\" (UniqueName: \"kubernetes.io/projected/6c0994d4-a4f2-4581-bdf3-e188ef354645-kube-api-access-bm46g\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b892b08-6ba5-42bc-84b6-85f34f7d8119-env-overrides\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.844698    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844725    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844748    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd1e2564-98b3-4035-819c-830ebcc71c13-serviceca\") pod \"node-ca-ft4xk\" (UID: \"cd1e2564-98b3-4035-819c-830ebcc71c13\") " pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.844768    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs podName:c7fde25b-60ca-405a-acf7-53e67418af9f nodeName:}" failed. No retries permitted until 2026-07-09 18:03:26.344745617 +0000 UTC m=+1.990739149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs") pod "network-metrics-daemon-5kshf" (UID: "c7fde25b-60ca-405a-acf7-53e67418af9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-modprobe-d\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844805    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-slash\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.844856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-slash\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a677a1da-0454-4e1e-9842-729623211e74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.845145    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrn7\" (UniqueName: \"kubernetes.io/projected/968af6c7-fb8f-4937-a490-2f31ecdae078-kube-api-access-lqrn7\") pod \"node-resolver-qpt8z\" (UID: \"968af6c7-fb8f-4937-a490-2f31ecdae078\") " pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.845216    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret podName:283ad84b-60d3-43ab-80e6-33da3e63bc77 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:26.345179921 +0000 UTC m=+1.991173438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret") pod "global-pull-secret-syncer-ql4td" (UID: "283ad84b-60d3-43ab-80e6-33da3e63bc77") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:25.846486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b892b08-6ba5-42bc-84b6-85f34f7d8119-env-overrides\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a677a1da-0454-4e1e-9842-729623211e74-cni-binary-copy\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-tuned\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845318    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-cni-binary-copy\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-sysconfig\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845385    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd1e2564-98b3-4035-819c-830ebcc71c13-serviceca\") pod \"node-ca-ft4xk\" (UID: \"cd1e2564-98b3-4035-819c-830ebcc71c13\") " pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845394    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-sysctl-d\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845429    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-var-lib-kubelet\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-run-ovn-kubernetes\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-sysctl-d\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845525    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-socket-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845552    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-sysconfig\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845555    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-var-lib-kubelet\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845591    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/968af6c7-fb8f-4937-a490-2f31ecdae078-hosts-file\") pod \"node-resolver-qpt8z\" (UID: \"968af6c7-fb8f-4937-a490-2f31ecdae078\") " pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845598    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c0994d4-a4f2-4581-bdf3-e188ef354645-tmp\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf3fb400-bb4c-49a8-96ba-a046356ba179-socket-dir\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.847087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845722    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c0994d4-a4f2-4581-bdf3-e188ef354645-var-lib-kubelet\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.847767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845768    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b892b08-6ba5-42bc-84b6-85f34f7d8119-host-run-ovn-kubernetes\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.847767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.845781    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a677a1da-0454-4e1e-9842-729623211e74-cni-binary-copy\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.847767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.847586    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b892b08-6ba5-42bc-84b6-85f34f7d8119-ovn-node-metrics-cert\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.847767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.847675    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a35e3606-7da3-4b0a-8c77-ebcef77fc51e-agent-certs\") pod \"konnectivity-agent-dgwzk\" (UID: \"a35e3606-7da3-4b0a-8c77-ebcef77fc51e\") " pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:25.847767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.847701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c0994d4-a4f2-4581-bdf3-e188ef354645-etc-tuned\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.847767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.847712    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c0994d4-a4f2-4581-bdf3-e188ef354645-tmp\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.852311 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.852287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh4hg\" (UniqueName: \"kubernetes.io/projected/c7fde25b-60ca-405a-acf7-53e67418af9f-kube-api-access-zh4hg\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:25.853029 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.853011    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g669d\" (UniqueName: \"kubernetes.io/projected/cd1e2564-98b3-4035-819c-830ebcc71c13-kube-api-access-g669d\") pod \"node-ca-ft4xk\" (UID: \"cd1e2564-98b3-4035-819c-830ebcc71c13\") " pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.853193 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.853172    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96x9\" (UniqueName: \"kubernetes.io/projected/6b892b08-6ba5-42bc-84b6-85f34f7d8119-kube-api-access-g96x9\") pod \"ovnkube-node-5s8lz\" (UID: \"6b892b08-6ba5-42bc-84b6-85f34f7d8119\") " pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:25.854971 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.854951    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w274c\" (UniqueName: \"kubernetes.io/projected/a677a1da-0454-4e1e-9842-729623211e74-kube-api-access-w274c\") pod \"multus-additional-cni-plugins-cbvg5\" (UID: \"a677a1da-0454-4e1e-9842-729623211e74\") " pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:25.863955 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.863712    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:25.863955 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.863735    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:25.863955 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.863747    2558 projected.go:196] Error preparing data for projected volume kube-api-access-hbzdg for pod openshift-network-diagnostics/network-check-target-tsz48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:25.863955 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:25.863808    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg podName:60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:26.363792856 +0000 UTC m=+2.009786375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hbzdg" (UniqueName: "kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg") pod "network-check-target-tsz48" (UID: "60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:25.866656 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.866447    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrn7\" (UniqueName: \"kubernetes.io/projected/968af6c7-fb8f-4937-a490-2f31ecdae078-kube-api-access-lqrn7\") pod \"node-resolver-qpt8z\" (UID: \"968af6c7-fb8f-4937-a490-2f31ecdae078\") " pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:25.866656 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.866594    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqm2\" (UniqueName: \"kubernetes.io/projected/bf3fb400-bb4c-49a8-96ba-a046356ba179-kube-api-access-fbqm2\") pod \"aws-ebs-csi-driver-node-zxp2c\" (UID: \"bf3fb400-bb4c-49a8-96ba-a046356ba179\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:25.867381 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.867354    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4vc\" (UniqueName: \"kubernetes.io/projected/559e8c07-add7-41ff-848b-fa5ed81dc791-kube-api-access-6j4vc\") pod \"iptables-alerter-hh2wt\" (UID: \"559e8c07-add7-41ff-848b-fa5ed81dc791\") " pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:25.867903 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.867882    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm46g\" (UniqueName: \"kubernetes.io/projected/6c0994d4-a4f2-4581-bdf3-e188ef354645-kube-api-access-bm46g\") pod \"tuned-nvswk\" (UID: \"6c0994d4-a4f2-4581-bdf3-e188ef354645\") " pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:25.887551 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:25.887522    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac32b18b87f72dcc3ed24191f7feefea.slice/crio-5d63df4a965c5adfa431f78f83f7e63c607a2b27e9bdf07beff7f81de517016e WatchSource:0}: Error finding container 5d63df4a965c5adfa431f78f83f7e63c607a2b27e9bdf07beff7f81de517016e: Status 404 returned error can't find the container with id 5d63df4a965c5adfa431f78f83f7e63c607a2b27e9bdf07beff7f81de517016e
Jul 09 18:03:25.887810 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:25.887789    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fa477e951a5bf596a004d58a807d25.slice/crio-f15ddbef8c6cb276cc22cc2708ce137220e43fa1eac56ff250edd8e86daffacc WatchSource:0}: Error finding container f15ddbef8c6cb276cc22cc2708ce137220e43fa1eac56ff250edd8e86daffacc: Status 404 returned error can't find the container with id f15ddbef8c6cb276cc22cc2708ce137220e43fa1eac56ff250edd8e86daffacc
Jul 09 18:03:25.893466 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.893446    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:03:25.894475 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.894462    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ft4xk"
Jul 09 18:03:25.903022 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:25.903001    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1e2564_98b3_4035_819c_830ebcc71c13.slice/crio-9f0e1872fb98bd1c3b2e74aca87111d13fc5464434640a5090c50e3545f0d7ac WatchSource:0}: Error finding container 9f0e1872fb98bd1c3b2e74aca87111d13fc5464434640a5090c50e3545f0d7ac: Status 404 returned error can't find the container with id 9f0e1872fb98bd1c3b2e74aca87111d13fc5464434640a5090c50e3545f0d7ac
Jul 09 18:03:25.945933 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.945911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-cni-dir\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.945933 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.945937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxhf5\" (UniqueName: \"kubernetes.io/projected/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-kube-api-access-gxhf5\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.945953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-system-cni-dir\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.945971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-var-lib-cni-bin\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.945985    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-run-multus-certs\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-os-release\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946031    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-run-k8s-cni-cncf-io\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-var-lib-cni-bin\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946047    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-system-cni-dir\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946047    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-cni-dir\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946069    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-run-multus-certs\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946081    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-run-k8s-cni-cncf-io\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-os-release\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946185    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-cni-binary-copy\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-var-lib-kubelet\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-daemon-config\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946293    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-var-lib-kubelet\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-cnibin\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-socket-dir-parent\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-hostroot\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-conf-dir\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946412    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-etc-kubernetes\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946451    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-run-netns\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946468    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-socket-dir-parent\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946474    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-var-lib-cni-multus\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946511    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-etc-kubernetes\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946536    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-var-lib-cni-multus\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-cnibin\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-hostroot\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946572    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-conf-dir\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946604    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-host-run-netns\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.946783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-cni-binary-copy\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.947307 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.946850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-multus-daemon-config\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:25.954666 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:25.954647    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxhf5\" (UniqueName: \"kubernetes.io/projected/b9c81c1a-2173-4d2c-a70a-96b9b1b323b3-kube-api-access-gxhf5\") pod \"multus-54sxh\" (UID: \"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3\") " pod="openshift-multus/multus-54sxh"
Jul 09 18:03:26.052297 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.052238    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cbvg5"
Jul 09 18:03:26.059867 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:26.059845    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda677a1da_0454_4e1e_9842_729623211e74.slice/crio-c848840398b799a3160f63b7709198c883b9ab3927dce76830de5cc77f6b9b26 WatchSource:0}: Error finding container c848840398b799a3160f63b7709198c883b9ab3927dce76830de5cc77f6b9b26: Status 404 returned error can't find the container with id c848840398b799a3160f63b7709198c883b9ab3927dce76830de5cc77f6b9b26
Jul 09 18:03:26.066949 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.066931    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hh2wt"
Jul 09 18:03:26.074152 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:26.074131    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559e8c07_add7_41ff_848b_fa5ed81dc791.slice/crio-e2342c8faf8ab9a8d0a5b47cd199fba1195b046b7d956bb08ab0d67a71337fcb WatchSource:0}: Error finding container e2342c8faf8ab9a8d0a5b47cd199fba1195b046b7d956bb08ab0d67a71337fcb: Status 404 returned error can't find the container with id e2342c8faf8ab9a8d0a5b47cd199fba1195b046b7d956bb08ab0d67a71337fcb
Jul 09 18:03:26.080229 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.080213    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:26.087157 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:26.087130    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35e3606_7da3_4b0a_8c77_ebcef77fc51e.slice/crio-09f8050efed4d308610cf1df7bc02033eed10d923d4f3499263284d8d8b44548 WatchSource:0}: Error finding container 09f8050efed4d308610cf1df7bc02033eed10d923d4f3499263284d8d8b44548: Status 404 returned error can't find the container with id 09f8050efed4d308610cf1df7bc02033eed10d923d4f3499263284d8d8b44548
Jul 09 18:03:26.098415 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.098397    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c"
Jul 09 18:03:26.106489 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:26.106470    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3fb400_bb4c_49a8_96ba_a046356ba179.slice/crio-755778d02128b88206b02cac660f90ba75ae0cf04a09e9fa6b6bd254b13c5f75 WatchSource:0}: Error finding container 755778d02128b88206b02cac660f90ba75ae0cf04a09e9fa6b6bd254b13c5f75: Status 404 returned error can't find the container with id 755778d02128b88206b02cac660f90ba75ae0cf04a09e9fa6b6bd254b13c5f75
Jul 09 18:03:26.118773 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.118757    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:26.125311 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:26.125292    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b892b08_6ba5_42bc_84b6_85f34f7d8119.slice/crio-7b562faf93529af361626140617bb4e038634a60f8e1b78f55b4f491635d1dfd WatchSource:0}: Error finding container 7b562faf93529af361626140617bb4e038634a60f8e1b78f55b4f491635d1dfd: Status 404 returned error can't find the container with id 7b562faf93529af361626140617bb4e038634a60f8e1b78f55b4f491635d1dfd
Jul 09 18:03:26.133122 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.133106    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvswk"
Jul 09 18:03:26.139880 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:26.139861    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0994d4_a4f2_4581_bdf3_e188ef354645.slice/crio-cf430aa7b05beaff5983f7128bcf0a489df50d22e939f7a3cec1d5349b18114a WatchSource:0}: Error finding container cf430aa7b05beaff5983f7128bcf0a489df50d22e939f7a3cec1d5349b18114a: Status 404 returned error can't find the container with id cf430aa7b05beaff5983f7128bcf0a489df50d22e939f7a3cec1d5349b18114a
Jul 09 18:03:26.157410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.157385    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qpt8z"
Jul 09 18:03:26.165014 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:26.164997    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968af6c7_fb8f_4937_a490_2f31ecdae078.slice/crio-61787050eb6b7ebd6c9e7a6ebcdf57ec519ed65d107da4e3ba5a9d3da90be2cf WatchSource:0}: Error finding container 61787050eb6b7ebd6c9e7a6ebcdf57ec519ed65d107da4e3ba5a9d3da90be2cf: Status 404 returned error can't find the container with id 61787050eb6b7ebd6c9e7a6ebcdf57ec519ed65d107da4e3ba5a9d3da90be2cf
Jul 09 18:03:26.200473 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.200450    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-54sxh"
Jul 09 18:03:26.207630 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:03:26.207416    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c81c1a_2173_4d2c_a70a_96b9b1b323b3.slice/crio-100ca37c74f91f9ef747c521cdec5f7247a0b38ac4c11ff2a2223fbe57083d57 WatchSource:0}: Error finding container 100ca37c74f91f9ef747c521cdec5f7247a0b38ac4c11ff2a2223fbe57083d57: Status 404 returned error can't find the container with id 100ca37c74f91f9ef747c521cdec5f7247a0b38ac4c11ff2a2223fbe57083d57
Jul 09 18:03:26.349568 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.349482    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:26.349568 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.349518    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:26.349792 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:26.349631    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:26.349792 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:26.349648    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:26.349792 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:26.349687    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret podName:283ad84b-60d3-43ab-80e6-33da3e63bc77 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:27.349673697 +0000 UTC m=+2.995667215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret") pod "global-pull-secret-syncer-ql4td" (UID: "283ad84b-60d3-43ab-80e6-33da3e63bc77") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:26.349792 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:26.349714    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs podName:c7fde25b-60ca-405a-acf7-53e67418af9f nodeName:}" failed. No retries permitted until 2026-07-09 18:03:27.349696652 +0000 UTC m=+2.995690186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs") pod "network-metrics-daemon-5kshf" (UID: "c7fde25b-60ca-405a-acf7-53e67418af9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:26.450247 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.450207    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:26.450430 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:26.450395    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:26.450430 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:26.450411    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:26.450430 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:26.450422    2558 projected.go:196] Error preparing data for projected volume kube-api-access-hbzdg for pod openshift-network-diagnostics/network-check-target-tsz48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:26.450632 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:26.450477    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg podName:60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:27.450455498 +0000 UTC m=+3.096449015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbzdg" (UniqueName: "kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg") pod "network-check-target-tsz48" (UID: "60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:26.508896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.508861    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:26.782805 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.782684    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:58:26 +0000 UTC" deadline="2028-02-14 19:37:38.069833694 +0000 UTC"
Jul 09 18:03:26.782805 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.782752    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14041h34m11.28708541s"
Jul 09 18:03:26.890258 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.890219    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:26.890446 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:26.890355    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:26.910903 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.910838    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal" event={"ID":"d2fa477e951a5bf596a004d58a807d25","Type":"ContainerStarted","Data":"f15ddbef8c6cb276cc22cc2708ce137220e43fa1eac56ff250edd8e86daffacc"}
Jul 09 18:03:26.921474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.921395    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal" event={"ID":"ac32b18b87f72dcc3ed24191f7feefea","Type":"ContainerStarted","Data":"5d63df4a965c5adfa431f78f83f7e63c607a2b27e9bdf07beff7f81de517016e"}
Jul 09 18:03:26.935411 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.935368    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qpt8z" event={"ID":"968af6c7-fb8f-4937-a490-2f31ecdae078","Type":"ContainerStarted","Data":"61787050eb6b7ebd6c9e7a6ebcdf57ec519ed65d107da4e3ba5a9d3da90be2cf"}
Jul 09 18:03:26.943706 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.943673    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c" event={"ID":"bf3fb400-bb4c-49a8-96ba-a046356ba179","Type":"ContainerStarted","Data":"755778d02128b88206b02cac660f90ba75ae0cf04a09e9fa6b6bd254b13c5f75"}
Jul 09 18:03:26.949735 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.949653    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dgwzk" event={"ID":"a35e3606-7da3-4b0a-8c77-ebcef77fc51e","Type":"ContainerStarted","Data":"09f8050efed4d308610cf1df7bc02033eed10d923d4f3499263284d8d8b44548"}
Jul 09 18:03:26.953491 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.953441    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbvg5" event={"ID":"a677a1da-0454-4e1e-9842-729623211e74","Type":"ContainerStarted","Data":"c848840398b799a3160f63b7709198c883b9ab3927dce76830de5cc77f6b9b26"}
Jul 09 18:03:26.956936 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.956906    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-54sxh" event={"ID":"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3","Type":"ContainerStarted","Data":"100ca37c74f91f9ef747c521cdec5f7247a0b38ac4c11ff2a2223fbe57083d57"}
Jul 09 18:03:26.964621 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.962754    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvswk" event={"ID":"6c0994d4-a4f2-4581-bdf3-e188ef354645","Type":"ContainerStarted","Data":"cf430aa7b05beaff5983f7128bcf0a489df50d22e939f7a3cec1d5349b18114a"}
Jul 09 18:03:26.969218 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.969189    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerStarted","Data":"7b562faf93529af361626140617bb4e038634a60f8e1b78f55b4f491635d1dfd"}
Jul 09 18:03:26.971677 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.971650    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hh2wt" event={"ID":"559e8c07-add7-41ff-848b-fa5ed81dc791","Type":"ContainerStarted","Data":"e2342c8faf8ab9a8d0a5b47cd199fba1195b046b7d956bb08ab0d67a71337fcb"}
Jul 09 18:03:26.984846 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:26.984815    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ft4xk" event={"ID":"cd1e2564-98b3-4035-819c-830ebcc71c13","Type":"ContainerStarted","Data":"9f0e1872fb98bd1c3b2e74aca87111d13fc5464434640a5090c50e3545f0d7ac"}
Jul 09 18:03:27.226304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:27.226033    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:27.358683 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:27.358642    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:27.358860 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:27.358699    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:27.358941 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.358869    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:27.358941 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.358935    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret podName:283ad84b-60d3-43ab-80e6-33da3e63bc77 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:29.358915821 +0000 UTC m=+5.004909346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret") pod "global-pull-secret-syncer-ql4td" (UID: "283ad84b-60d3-43ab-80e6-33da3e63bc77") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:27.359062 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.359000    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:27.359062 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.359032    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs podName:c7fde25b-60ca-405a-acf7-53e67418af9f nodeName:}" failed. No retries permitted until 2026-07-09 18:03:29.359020643 +0000 UTC m=+5.005014167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs") pod "network-metrics-daemon-5kshf" (UID: "c7fde25b-60ca-405a-acf7-53e67418af9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:27.460102 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:27.460062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:27.460278 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.460256    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:27.460351 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.460286    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:27.460351 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.460300    2558 projected.go:196] Error preparing data for projected volume kube-api-access-hbzdg for pod openshift-network-diagnostics/network-check-target-tsz48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:27.460502 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.460358    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg podName:60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:29.460338346 +0000 UTC m=+5.106331867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbzdg" (UniqueName: "kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg") pod "network-check-target-tsz48" (UID: "60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:27.783114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:27.783047    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:58:26 +0000 UTC" deadline="2027-12-25 20:36:05.294671692 +0000 UTC"
Jul 09 18:03:27.783114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:27.783091    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12818h32m37.511586383s"
Jul 09 18:03:27.887196 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:27.887167    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:27.887298 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.887273    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:27.887538 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:27.887516    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:27.887676 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:27.887652    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:28.890461 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:28.890374    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:28.890946 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:28.890554    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:29.377525 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:29.377431    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:29.377525 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:29.377494    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:29.377894 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.377682    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:29.377894 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.377751    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret podName:283ad84b-60d3-43ab-80e6-33da3e63bc77 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:33.377730208 +0000 UTC m=+9.023723740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret") pod "global-pull-secret-syncer-ql4td" (UID: "283ad84b-60d3-43ab-80e6-33da3e63bc77") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:29.377894 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.377835    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:29.377894 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.377873    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs podName:c7fde25b-60ca-405a-acf7-53e67418af9f nodeName:}" failed. No retries permitted until 2026-07-09 18:03:33.37785951 +0000 UTC m=+9.023853029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs") pod "network-metrics-daemon-5kshf" (UID: "c7fde25b-60ca-405a-acf7-53e67418af9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:29.478196 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:29.478158    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:29.478404 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.478343    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:29.478404 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.478386    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:29.478404 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.478401    2558 projected.go:196] Error preparing data for projected volume kube-api-access-hbzdg for pod openshift-network-diagnostics/network-check-target-tsz48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:29.478588 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.478461    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg podName:60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:33.478441361 +0000 UTC m=+9.124434884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbzdg" (UniqueName: "kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg") pod "network-check-target-tsz48" (UID: "60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:29.887382 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:29.887330    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:29.887552 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.887491    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:29.887871 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:29.887853    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:29.887971 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:29.887937    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:30.896482 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:30.896446    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:30.896847 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:30.896583    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:31.888679 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:31.888183    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:31.888679 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:31.888345    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:31.888679 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:31.888358    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:31.888679 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:31.888487    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:32.887928 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:32.887897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:32.888298 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:32.888020    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:33.410287 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:33.410245    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:33.410474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:33.410295    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:33.410474 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.410433    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:33.410599 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.410507    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret podName:283ad84b-60d3-43ab-80e6-33da3e63bc77 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:41.410488043 +0000 UTC m=+17.056481574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret") pod "global-pull-secret-syncer-ql4td" (UID: "283ad84b-60d3-43ab-80e6-33da3e63bc77") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:33.410599 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.410433    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:33.410599 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.410589    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs podName:c7fde25b-60ca-405a-acf7-53e67418af9f nodeName:}" failed. No retries permitted until 2026-07-09 18:03:41.410571041 +0000 UTC m=+17.056564560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs") pod "network-metrics-daemon-5kshf" (UID: "c7fde25b-60ca-405a-acf7-53e67418af9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:33.511610 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:33.511029    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:33.511610 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.511214    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:33.511610 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.511238    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:33.511610 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.511250    2558 projected.go:196] Error preparing data for projected volume kube-api-access-hbzdg for pod openshift-network-diagnostics/network-check-target-tsz48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:33.511610 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.511296    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg podName:60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:41.511284073 +0000 UTC m=+17.157277590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbzdg" (UniqueName: "kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg") pod "network-check-target-tsz48" (UID: "60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:33.887686 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:33.887614    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:33.887827 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.887796    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:33.888152 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:33.888136    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:33.888518 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:33.888219    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:34.889212 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:34.889180    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:34.889630 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:34.889300    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:35.887984 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:35.887955    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:35.887984 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:35.887982    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:35.888222 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:35.888081    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:35.888222 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:35.888201    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:36.890774 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:36.890285    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:36.890774 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:36.890430    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:37.887761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:37.887721    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:37.887954 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:37.887722    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:37.887954 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:37.887873    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:37.887954 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:37.887913    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:38.887624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:38.887579    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:38.888085 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:38.887710    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:39.887580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:39.887542    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:39.887751 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:39.887589    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:39.887751 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:39.887682    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:39.888163 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:39.888082    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:40.887531 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:40.887495    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:40.887728 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:40.887621    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:41.472981 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:41.472942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:41.472981 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:41.472983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:41.473563 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.473067    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:41.473563 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.473106    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:41.473563 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.473142    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs podName:c7fde25b-60ca-405a-acf7-53e67418af9f nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.473122575 +0000 UTC m=+33.119116097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs") pod "network-metrics-daemon-5kshf" (UID: "c7fde25b-60ca-405a-acf7-53e67418af9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:41.473563 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.473158    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret podName:283ad84b-60d3-43ab-80e6-33da3e63bc77 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.473151163 +0000 UTC m=+33.119144680 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret") pod "global-pull-secret-syncer-ql4td" (UID: "283ad84b-60d3-43ab-80e6-33da3e63bc77") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:41.573561 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:41.573531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:41.573751 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.573721    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:41.573751 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.573744    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:41.573866 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.573753    2558 projected.go:196] Error preparing data for projected volume kube-api-access-hbzdg for pod openshift-network-diagnostics/network-check-target-tsz48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:41.573866 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.573818    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg podName:60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:57.573789258 +0000 UTC m=+33.219782778 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbzdg" (UniqueName: "kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg") pod "network-check-target-tsz48" (UID: "60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:41.888009 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:41.887929    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:41.888233 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.888069    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:41.888233 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:41.888131    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:41.888309 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:41.888260    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:42.887404 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:42.887375    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:42.887855 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:42.887502    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:43.887369 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:43.887312    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:43.887369 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:43.887358    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:43.887901 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:43.887467    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:43.887901 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:43.887588    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:44.888344 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:44.888290    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:44.888779 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:44.888438    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:45.887994 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:45.887956    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:45.888181 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:45.888007    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:45.888181 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:45.888124    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:45.888292 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:45.888263    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:46.887271 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:46.887244    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:46.887618 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:46.887390    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:47.032616 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:47.032586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal" event={"ID":"d2fa477e951a5bf596a004d58a807d25","Type":"ContainerStarted","Data":"0eb4deaf334c63caf057e22563852cd5a0a1512f5cf508418bd07d6c664f7241"}
Jul 09 18:03:47.888496 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:47.888089    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:47.889203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:47.888150    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:47.889203 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:47.888589    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:47.889203 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:47.888652    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:48.036677 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.036645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dgwzk" event={"ID":"a35e3606-7da3-4b0a-8c77-ebcef77fc51e","Type":"ContainerStarted","Data":"5211d1eba8cccadecb7a72237821e8abbf294e61111b62d5089e85531f622924"}
Jul 09 18:03:48.038170 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.038141    2558 generic.go:356] "Generic (PLEG): container finished" podID="a677a1da-0454-4e1e-9842-729623211e74" containerID="02639db4672dffd3786dc1f7f0e06900776d6a2b712d43b53e86aa1a94be582c" exitCode=0
Jul 09 18:03:48.038287 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.038189    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbvg5" event={"ID":"a677a1da-0454-4e1e-9842-729623211e74","Type":"ContainerDied","Data":"02639db4672dffd3786dc1f7f0e06900776d6a2b712d43b53e86aa1a94be582c"}
Jul 09 18:03:48.040311 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.040279    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-54sxh" event={"ID":"b9c81c1a-2173-4d2c-a70a-96b9b1b323b3","Type":"ContainerStarted","Data":"d33904f9f3882a9a24e2835af436a3f36f3cb8a6c3ced587b9553050f1153a81"}
Jul 09 18:03:48.041392 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.041371    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvswk" event={"ID":"6c0994d4-a4f2-4581-bdf3-e188ef354645","Type":"ContainerStarted","Data":"c0bfcabc1730e8aac9a86ea964e2945e334b6dcad4a0d23e569aaeac59646315"}
Jul 09 18:03:48.043424 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.043408    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:03:48.043675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.043659    2558 generic.go:356] "Generic (PLEG): container finished" podID="6b892b08-6ba5-42bc-84b6-85f34f7d8119" containerID="1abff195113e3e8f1a9dcbdf24c8f32583d46d21fd191a25526f83c487841e24" exitCode=1
Jul 09 18:03:48.043730 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.043705    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerStarted","Data":"2aaa55e54af569330767885732aaab4d4fe2834d71932022e7c189b444ddcde2"}
Jul 09 18:03:48.043764 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.043750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerStarted","Data":"6913410c31564fe66e4013cf88fe22e082df14b2f3675755a62468795b0e79dc"}
Jul 09 18:03:48.043796 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.043779    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerStarted","Data":"1661ba5fc9a17de090d83fb336a5495b44c9658537411728114fa3e6f091774a"}
Jul 09 18:03:48.043827 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.043808    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerStarted","Data":"80bdfbb8706fdd371fd5d4ee9346838de1456ec311a660255d71eba3c07c320a"}
Jul 09 18:03:48.043859 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.043834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerDied","Data":"1abff195113e3e8f1a9dcbdf24c8f32583d46d21fd191a25526f83c487841e24"}
Jul 09 18:03:48.043890 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.043861    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerStarted","Data":"46ce3165024cabf7bc63f4a3d71c2c3b777a02bada731a1ca412d72ea89448a8"}
Jul 09 18:03:48.045010 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.044990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ft4xk" event={"ID":"cd1e2564-98b3-4035-819c-830ebcc71c13","Type":"ContainerStarted","Data":"0e8e7fd1c7584a57db1f266789811dd0ebf642c50b41ae9dd2143853571b8c66"}
Jul 09 18:03:48.046651 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.046631    2558 generic.go:356] "Generic (PLEG): container finished" podID="ac32b18b87f72dcc3ed24191f7feefea" containerID="fe88af5338527b757940590c5cfccbb9c079fbc612d13349900826062c9aca82" exitCode=0
Jul 09 18:03:48.046722 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.046694    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal" event={"ID":"ac32b18b87f72dcc3ed24191f7feefea","Type":"ContainerDied","Data":"fe88af5338527b757940590c5cfccbb9c079fbc612d13349900826062c9aca82"}
Jul 09 18:03:48.048111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.048092    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qpt8z" event={"ID":"968af6c7-fb8f-4937-a490-2f31ecdae078","Type":"ContainerStarted","Data":"65b73c31ffb722de2c9f04993bec54c665bfdd96cfebb50ab5a77092c74c0a7f"}
Jul 09 18:03:48.049468 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.049437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c" event={"ID":"bf3fb400-bb4c-49a8-96ba-a046356ba179","Type":"ContainerStarted","Data":"41f6d7a4085fdad9a48acdea786a99d70e26cda382a1205c040045a75a6f5451"}
Jul 09 18:03:48.063418 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.063361    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-224.ec2.internal" podStartSLOduration=22.063348256 podStartE2EDuration="22.063348256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:47.055577071 +0000 UTC m=+22.701570611" watchObservedRunningTime="2026-07-09 18:03:48.063348256 +0000 UTC m=+23.709341773"
Jul 09 18:03:48.063837 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.063816    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dgwzk" podStartSLOduration=2.467648501 podStartE2EDuration="23.063807791s" totalImagesPullingTime="20.59615929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:48.063483412 +0000 UTC m=+23.709476952" watchObservedRunningTime="2026-07-09 18:03:48.063807791 +0000 UTC m=+23.709801330"
Jul 09 18:03:48.085190 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.085146    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ft4xk" podStartSLOduration=2.304916189 podStartE2EDuration="23.085131907s" totalImagesPullingTime="20.780215718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:48.084929424 +0000 UTC m=+23.730922964" watchObservedRunningTime="2026-07-09 18:03:48.085131907 +0000 UTC m=+23.731125446"
Jul 09 18:03:48.128157 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.128135    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:48.146163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.146109    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-54sxh" podStartSLOduration=2.187655075 podStartE2EDuration="23.146094897s" totalImagesPullingTime="20.958439822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:48.111156088 +0000 UTC m=+23.757149627" watchObservedRunningTime="2026-07-09 18:03:48.146094897 +0000 UTC m=+23.792088436"
Jul 09 18:03:48.195011 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.194893    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nvswk" podStartSLOduration=2.563515385 podStartE2EDuration="23.194874934s" totalImagesPullingTime="20.631359549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:48.171687322 +0000 UTC m=+23.817680860" watchObservedRunningTime="2026-07-09 18:03:48.194874934 +0000 UTC m=+23.840868473"
Jul 09 18:03:48.228916 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.228878    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qpt8z" podStartSLOduration=2.61984654 podStartE2EDuration="23.228862983s" totalImagesPullingTime="20.609016443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:48.195159716 +0000 UTC m=+23.841153247" watchObservedRunningTime="2026-07-09 18:03:48.228862983 +0000 UTC m=+23.874856522"
Jul 09 18:03:48.887677 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:48.887649    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:48.887839 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:48.887763    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:49.012767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.012741    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:03:49.056154 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.056120    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal" event={"ID":"ac32b18b87f72dcc3ed24191f7feefea","Type":"ContainerStarted","Data":"272a3684050d8d878355c64d26ab445a4ba62ff4af87e72f08d81ebcbf41266f"}
Jul 09 18:03:49.057849 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.057826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c" event={"ID":"bf3fb400-bb4c-49a8-96ba-a046356ba179","Type":"ContainerStarted","Data":"bf9f32ddefcf3084a639fb5ac77c59a19d9197f1d52c225e89c8e0f1f19a4fb8"}
Jul 09 18:03:49.059088 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.059059    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hh2wt" event={"ID":"559e8c07-add7-41ff-848b-fa5ed81dc791","Type":"ContainerStarted","Data":"1c0a95b6a73c6c5f5ab045eea4528c2a697ebe9e5d92f29ae42ec290b8588128"}
Jul 09 18:03:49.076012 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.075971    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-224.ec2.internal" podStartSLOduration=23.075958623 podStartE2EDuration="23.075958623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:49.075395409 +0000 UTC m=+24.721388950" watchObservedRunningTime="2026-07-09 18:03:49.075958623 +0000 UTC m=+24.721952162"
Jul 09 18:03:49.806908 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.806802    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:03:49.012761942Z","UUID":"aa427988-f7cf-447b-aa56-406abc8c9029","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:03:49.808417 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.808395    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:03:49.808542 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.808438    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:03:49.887811 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.887775    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:49.887811 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:49.887799    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:49.888044 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:49.887907    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:49.888044 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:49.888030    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:50.061900 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:50.061471    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:50.062228 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:50.062180    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:50.066044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:50.066019    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:03:50.067541 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:50.067471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerStarted","Data":"47b700bf718139744b629213bb7f4c92b4c4ec28c49b85376bf705bfea52a9a9"}
Jul 09 18:03:50.068300 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:50.068279    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dgwzk"
Jul 09 18:03:50.081109 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:50.081055    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hh2wt" podStartSLOduration=4.38683821 podStartE2EDuration="25.081025443s" totalImagesPullingTime="20.694187233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:49.093367522 +0000 UTC m=+24.739361063" watchObservedRunningTime="2026-07-09 18:03:50.081025443 +0000 UTC m=+25.727018983"
Jul 09 18:03:50.887592 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:50.887555    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:50.887771 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:50.887677    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:51.071068 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:51.070985    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c" event={"ID":"bf3fb400-bb4c-49a8-96ba-a046356ba179","Type":"ContainerStarted","Data":"f2cb4e5067a3a21741b02d3f04fe93dc196530bef7eabe62bccabbc1a19e7a38"}
Jul 09 18:03:51.887626 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:51.887600    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:51.887792 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:51.887604    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:51.887792 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:51.887698    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:51.887792 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:51.887785    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:52.887637 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:52.887413    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:52.888209 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:52.887728    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:53.079188 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:53.079163    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:03:53.079640 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:53.079608    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerStarted","Data":"5ffb40a6d734e92e6add3ada787827632e704e22f10c7d8b40db5cae0d1e0d5b"}
Jul 09 18:03:53.079934 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:53.079914    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:53.080014 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:53.079992    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:53.080144 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:53.080123    2558 scope.go:117] "RemoveContainer" containerID="1abff195113e3e8f1a9dcbdf24c8f32583d46d21fd191a25526f83c487841e24"
Jul 09 18:03:53.098416 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:53.098390    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:53.115393 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:53.115319    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxp2c" podStartSLOduration=4.372116239 podStartE2EDuration="28.115300496s" totalImagesPullingTime="23.743184257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:51.09349315 +0000 UTC m=+26.739486690" watchObservedRunningTime="2026-07-09 18:03:53.115300496 +0000 UTC m=+28.761294037"
Jul 09 18:03:53.888229 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:53.888194    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:53.888229 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:53.888223    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:53.888793 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:53.888319    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:53.888793 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:53.888413    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:54.342284 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:54.342096    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kshf"]
Jul 09 18:03:54.342460 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:54.342384    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:54.342508 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:54.342485    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:54.344599 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:54.344573    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tsz48"]
Jul 09 18:03:54.344724 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:54.344701    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:54.344828 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:54.344803    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:54.356374 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:54.356349    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ql4td"]
Jul 09 18:03:54.356483 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:54.356440    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:54.356582 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:54.356558    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:55.086111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.086075    2558 generic.go:356] "Generic (PLEG): container finished" podID="a677a1da-0454-4e1e-9842-729623211e74" containerID="63a4a8ac0cd38782768cd52ee6368c3ef7e79c9fea5632ae6c9ef8f7c7b1fcf3" exitCode=0
Jul 09 18:03:55.086615 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.086159    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbvg5" event={"ID":"a677a1da-0454-4e1e-9842-729623211e74","Type":"ContainerDied","Data":"63a4a8ac0cd38782768cd52ee6368c3ef7e79c9fea5632ae6c9ef8f7c7b1fcf3"}
Jul 09 18:03:55.089991 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.089974    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:03:55.090370 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.090350    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" event={"ID":"6b892b08-6ba5-42bc-84b6-85f34f7d8119","Type":"ContainerStarted","Data":"b01dd1fde58d0722fbc0f7f0e97ed30c3a40c66921c4cd7cdd804e5bb7901882"}
Jul 09 18:03:55.090520 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.090507    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:55.106569 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.106549    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:03:55.146375 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.146308    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" podStartSLOduration=9.170182992 podStartE2EDuration="30.146294003s" totalImagesPullingTime="20.976111011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:55.145965485 +0000 UTC m=+30.791959037" watchObservedRunningTime="2026-07-09 18:03:55.146294003 +0000 UTC m=+30.792287520"
Jul 09 18:03:55.887719 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.887690    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:55.887719 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.887721    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:55.887965 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:55.887796    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:55.887965 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:55.887808    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:55.887965 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:55.887912    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:55.888117 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:55.888034    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:57.096819 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:57.096780    2558 generic.go:356] "Generic (PLEG): container finished" podID="a677a1da-0454-4e1e-9842-729623211e74" containerID="5a2b0f162907391c48194d6c8b91f6d24bb60726eb44c5d871b55328cfff3647" exitCode=0
Jul 09 18:03:57.097711 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:57.096864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbvg5" event={"ID":"a677a1da-0454-4e1e-9842-729623211e74","Type":"ContainerDied","Data":"5a2b0f162907391c48194d6c8b91f6d24bb60726eb44c5d871b55328cfff3647"}
Jul 09 18:03:57.108670 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:57.108632    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz" podUID="6b892b08-6ba5-42bc-84b6-85f34f7d8119" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 09 18:03:57.491039 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:57.491007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:57.491039 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:57.491041    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:57.491218 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.491135    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:57.491218 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.491141    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:57.491218 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.491186    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret podName:283ad84b-60d3-43ab-80e6-33da3e63bc77 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:29.491173084 +0000 UTC m=+65.137166603 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret") pod "global-pull-secret-syncer-ql4td" (UID: "283ad84b-60d3-43ab-80e6-33da3e63bc77") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:57.491218 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.491200    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs podName:c7fde25b-60ca-405a-acf7-53e67418af9f nodeName:}" failed. No retries permitted until 2026-07-09 18:04:29.491194349 +0000 UTC m=+65.137187865 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs") pod "network-metrics-daemon-5kshf" (UID: "c7fde25b-60ca-405a-acf7-53e67418af9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:57.592158 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:57.592122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:57.592349 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.592272    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:57.592349 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.592290    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:57.592349 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.592300    2558 projected.go:196] Error preparing data for projected volume kube-api-access-hbzdg for pod openshift-network-diagnostics/network-check-target-tsz48: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:57.592476 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.592371    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg podName:60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:29.592357949 +0000 UTC m=+65.238351466 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbzdg" (UniqueName: "kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg") pod "network-check-target-tsz48" (UID: "60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:57.888185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:57.888109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:57.888367 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:57.888109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:57.888367 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.888227    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ql4td" podUID="283ad84b-60d3-43ab-80e6-33da3e63bc77"
Jul 09 18:03:57.888367 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:57.888109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:57.888367 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.888321    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kshf" podUID="c7fde25b-60ca-405a-acf7-53e67418af9f"
Jul 09 18:03:57.888522 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:57.888411    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsz48" podUID="60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00"
Jul 09 18:03:58.101376 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:58.101347    2558 generic.go:356] "Generic (PLEG): container finished" podID="a677a1da-0454-4e1e-9842-729623211e74" containerID="e00948265687a673309a286c0da37b04e1fa09d55b71ac8f4b1ee0873ddcb942" exitCode=0
Jul 09 18:03:58.101825 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:58.101418    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbvg5" event={"ID":"a677a1da-0454-4e1e-9842-729623211e74","Type":"ContainerDied","Data":"e00948265687a673309a286c0da37b04e1fa09d55b71ac8f4b1ee0873ddcb942"}
Jul 09 18:03:58.184431 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:58.184362    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qpt8z_968af6c7-fb8f-4937-a490-2f31ecdae078/dns-node-resolver/0.log"
Jul 09 18:03:58.765340 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:58.765095    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ft4xk_cd1e2564-98b3-4035-819c-830ebcc71c13/node-ca/0.log"
Jul 09 18:03:59.636774 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.636742    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-224.ec2.internal" event="NodeReady"
Jul 09 18:03:59.637173 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.636904    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:03:59.691093 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.691056    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"]
Jul 09 18:03:59.693587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.693563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.700191 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.700169    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:03:59.700756 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.700587    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:03:59.700962 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.700944    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:03:59.701242 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.701223    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pxfrt\""
Jul 09 18:03:59.709030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.709012    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:03:59.710663 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.710641    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9gflx"]
Jul 09 18:03:59.714704 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.714522    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9gflx"
Jul 09 18:03:59.715085 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.715062    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"]
Jul 09 18:03:59.715859 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.715842    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kmvpf"]
Jul 09 18:03:59.720843 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.720789    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qqkr6\""
Jul 09 18:03:59.720944 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.720882    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.720944 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.720934    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:03:59.721101 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.721082    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:03:59.721295 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.721114    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:03:59.726227 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.726208    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f5z8w\""
Jul 09 18:03:59.726511 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.726496    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:03:59.726826 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.726811    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:03:59.726907 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.726813    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:03:59.730068 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.730048    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-b8vj2"]
Jul 09 18:03:59.732722 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.732705    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:03:59.733903 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.733885    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9gflx"]
Jul 09 18:03:59.733976 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.733961    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.738053 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.738005    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 09 18:03:59.738169 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.738151    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c7bwg\""
Jul 09 18:03:59.738827 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.738810    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 09 18:03:59.740816 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.740797    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmvpf"]
Jul 09 18:03:59.742750 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.742733    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 09 18:03:59.743126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.743112    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 09 18:03:59.748824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.748803    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-b8vj2"]
Jul 09 18:03:59.749345 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.749310    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b2gwz"]
Jul 09 18:03:59.752902 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.752884    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:03:59.757855 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.757835    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:03:59.757951 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.757926    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ln8j9\""
Jul 09 18:03:59.758016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.757930    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:03:59.766830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.766808    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b2gwz"]
Jul 09 18:03:59.807746 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.807559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.807746 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.807641    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knv98\" (UniqueName: \"kubernetes.io/projected/f446a6a5-d53a-4147-a454-18dc8917eadd-kube-api-access-knv98\") pod \"ingress-canary-9gflx\" (UID: \"f446a6a5-d53a-4147-a454-18dc8917eadd\") " pod="openshift-ingress-canary/ingress-canary-9gflx"
Jul 09 18:03:59.807746 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.807670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-image-registry-private-configuration\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.807746 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.807706    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-trusted-ca\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.808055 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.807780    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3be24e31-d32d-4306-bd83-bb3fca6b64b5-signing-key\") pod \"service-ca-79f68b9f4f-b8vj2\" (UID: \"3be24e31-d32d-4306-bd83-bb3fca6b64b5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.808055 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.807809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-certificates\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.808055 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.807834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-bound-sa-token\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.808055 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.807854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3be24e31-d32d-4306-bd83-bb3fca6b64b5-signing-cabundle\") pod \"service-ca-79f68b9f4f-b8vj2\" (UID: \"3be24e31-d32d-4306-bd83-bb3fca6b64b5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.808055 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.807900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5507b99-2a64-4f58-a456-4c80b25edf93-ca-trust-extracted\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.808055 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.808007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8pl4\" (UniqueName: \"kubernetes.io/projected/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-api-access-r8pl4\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.808055 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.808051    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.808412 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.808080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b472c8fe-f511-4965-91fa-237ad1fd8f6d-crio-socket\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.808412 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.808109    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f446a6a5-d53a-4147-a454-18dc8917eadd-cert\") pod \"ingress-canary-9gflx\" (UID: \"f446a6a5-d53a-4147-a454-18dc8917eadd\") " pod="openshift-ingress-canary/ingress-canary-9gflx"
Jul 09 18:03:59.808412 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.808135    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-installation-pull-secrets\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.808412 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.808185    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.808412 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.808216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4zlg\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-kube-api-access-w4zlg\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.808412 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.808270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8cgb\" (UniqueName: \"kubernetes.io/projected/3be24e31-d32d-4306-bd83-bb3fca6b64b5-kube-api-access-d8cgb\") pod \"service-ca-79f68b9f4f-b8vj2\" (UID: \"3be24e31-d32d-4306-bd83-bb3fca6b64b5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.808412 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.808298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b472c8fe-f511-4965-91fa-237ad1fd8f6d-data-volume\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.887816 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.887732    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:03:59.887977 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.887832    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:03:59.887977 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.887841    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:03:59.898668 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.898645    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:03:59.898880 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.898865    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:03:59.899569 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.899547    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:03:59.899690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.899599    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:03:59.899864 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.899846    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cmkm4\""
Jul 09 18:03:59.900907 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.900891    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hzqhq\""
Jul 09 18:03:59.908935 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.908916    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b472c8fe-f511-4965-91fa-237ad1fd8f6d-data-volume\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.909026 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.908941    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.909026 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.908981    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec64976d-0e4f-4c30-9203-b000c1597408-tmp-dir\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:03:59.909026 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knv98\" (UniqueName: \"kubernetes.io/projected/f446a6a5-d53a-4147-a454-18dc8917eadd-kube-api-access-knv98\") pod \"ingress-canary-9gflx\" (UID: \"f446a6a5-d53a-4147-a454-18dc8917eadd\") " pod="openshift-ingress-canary/ingress-canary-9gflx"
Jul 09 18:03:59.909200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909034    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-image-registry-private-configuration\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.909200 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:59.909050    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:03:59.909200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-trusted-ca\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.909200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3be24e31-d32d-4306-bd83-bb3fca6b64b5-signing-key\") pod \"service-ca-79f68b9f4f-b8vj2\" (UID: \"3be24e31-d32d-4306-bd83-bb3fca6b64b5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.909200 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:59.909118    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:00.409097411 +0000 UTC m=+36.055090945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : configmap "kube-rbac-proxy" not found
Jul 09 18:03:59.909200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-certificates\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.909200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909188    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-bound-sa-token\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909220    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3be24e31-d32d-4306-bd83-bb3fca6b64b5-signing-cabundle\") pod \"service-ca-79f68b9f4f-b8vj2\" (UID: \"3be24e31-d32d-4306-bd83-bb3fca6b64b5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909249    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909278    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b472c8fe-f511-4965-91fa-237ad1fd8f6d-data-volume\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5507b99-2a64-4f58-a456-4c80b25edf93-ca-trust-extracted\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pl4\" (UniqueName: \"kubernetes.io/projected/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-api-access-r8pl4\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96qvh\" (UniqueName: \"kubernetes.io/projected/ec64976d-0e4f-4c30-9203-b000c1597408-kube-api-access-96qvh\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909403    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b472c8fe-f511-4965-91fa-237ad1fd8f6d-crio-socket\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f446a6a5-d53a-4147-a454-18dc8917eadd-cert\") pod \"ingress-canary-9gflx\" (UID: \"f446a6a5-d53a-4147-a454-18dc8917eadd\") " pod="openshift-ingress-canary/ingress-canary-9gflx"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909482    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-installation-pull-secrets\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:59.909537    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:03:59.909564 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:59.909552    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5fdd7ff6-khzmb: secret "image-registry-tls" not found
Jul 09 18:03:59.910114 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:59.909603    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls podName:f5507b99-2a64-4f58-a456-4c80b25edf93 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:00.409587295 +0000 UTC m=+36.055580836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls") pod "image-registry-5c5fdd7ff6-khzmb" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93") : secret "image-registry-tls" not found
Jul 09 18:03:59.910114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909540    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4zlg\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-kube-api-access-w4zlg\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.910114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909652    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec64976d-0e4f-4c30-9203-b000c1597408-config-volume\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:03:59.910114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8cgb\" (UniqueName: \"kubernetes.io/projected/3be24e31-d32d-4306-bd83-bb3fca6b64b5-kube-api-access-d8cgb\") pod \"service-ca-79f68b9f4f-b8vj2\" (UID: \"3be24e31-d32d-4306-bd83-bb3fca6b64b5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.910114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909834    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5507b99-2a64-4f58-a456-4c80b25edf93-ca-trust-extracted\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.910114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909917    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b472c8fe-f511-4965-91fa-237ad1fd8f6d-crio-socket\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.910114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.909937    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-certificates\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.910114 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:59.909975    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:03:59.910114 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:03:59.910020    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:00.410006344 +0000 UTC m=+36.055999862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:03:59.910536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.910180    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-trusted-ca\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.910536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.910437    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3be24e31-d32d-4306-bd83-bb3fca6b64b5-signing-cabundle\") pod \"service-ca-79f68b9f4f-b8vj2\" (UID: \"3be24e31-d32d-4306-bd83-bb3fca6b64b5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.913495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.913470    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-installation-pull-secrets\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.913595 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.913500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-image-registry-private-configuration\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.913645 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.913626    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f446a6a5-d53a-4147-a454-18dc8917eadd-cert\") pod \"ingress-canary-9gflx\" (UID: \"f446a6a5-d53a-4147-a454-18dc8917eadd\") " pod="openshift-ingress-canary/ingress-canary-9gflx"
Jul 09 18:03:59.913677 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.913626    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3be24e31-d32d-4306-bd83-bb3fca6b64b5-signing-key\") pod \"service-ca-79f68b9f4f-b8vj2\" (UID: \"3be24e31-d32d-4306-bd83-bb3fca6b64b5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.921397 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.921378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-bound-sa-token\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:03:59.926646 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.926615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8cgb\" (UniqueName: \"kubernetes.io/projected/3be24e31-d32d-4306-bd83-bb3fca6b64b5-kube-api-access-d8cgb\") pod \"service-ca-79f68b9f4f-b8vj2\" (UID: \"3be24e31-d32d-4306-bd83-bb3fca6b64b5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:03:59.927081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.927065    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knv98\" (UniqueName: \"kubernetes.io/projected/f446a6a5-d53a-4147-a454-18dc8917eadd-kube-api-access-knv98\") pod \"ingress-canary-9gflx\" (UID: \"f446a6a5-d53a-4147-a454-18dc8917eadd\") " pod="openshift-ingress-canary/ingress-canary-9gflx"
Jul 09 18:03:59.927260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.927236    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pl4\" (UniqueName: \"kubernetes.io/projected/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-api-access-r8pl4\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:03:59.927556 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:03:59.927539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4zlg\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-kube-api-access-w4zlg\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:00.010023 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.009982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec64976d-0e4f-4c30-9203-b000c1597408-config-volume\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:00.010201 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.010059    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec64976d-0e4f-4c30-9203-b000c1597408-tmp-dir\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:00.010201 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.010105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:00.010201 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.010125    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96qvh\" (UniqueName: \"kubernetes.io/projected/ec64976d-0e4f-4c30-9203-b000c1597408-kube-api-access-96qvh\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:00.010385 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.010240    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:00.010385 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.010317    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls podName:ec64976d-0e4f-4c30-9203-b000c1597408 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:00.510297524 +0000 UTC m=+36.156291058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls") pod "dns-default-b2gwz" (UID: "ec64976d-0e4f-4c30-9203-b000c1597408") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:00.010521 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.010500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec64976d-0e4f-4c30-9203-b000c1597408-tmp-dir\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:00.010649 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.010621    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec64976d-0e4f-4c30-9203-b000c1597408-config-volume\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:00.020195 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.020174    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96qvh\" (UniqueName: \"kubernetes.io/projected/ec64976d-0e4f-4c30-9203-b000c1597408-kube-api-access-96qvh\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:00.028078 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.028056    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9gflx"
Jul 09 18:04:00.043350 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.043308    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2"
Jul 09 18:04:00.192506 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.192474    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9gflx"]
Jul 09 18:04:00.197586 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:00.197547    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf446a6a5_d53a_4147_a454_18dc8917eadd.slice/crio-c7d87e76383eba7e06b8939d6f142f8ca0d37695b7835b70723e6c48b933ee7d WatchSource:0}: Error finding container c7d87e76383eba7e06b8939d6f142f8ca0d37695b7835b70723e6c48b933ee7d: Status 404 returned error can't find the container with id c7d87e76383eba7e06b8939d6f142f8ca0d37695b7835b70723e6c48b933ee7d
Jul 09 18:04:00.197688 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.197597    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-b8vj2"]
Jul 09 18:04:00.201073 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:00.201038    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be24e31_d32d_4306_bd83_bb3fca6b64b5.slice/crio-c3eaec711be3d1a7adb1a18ccbaada3f16ebb87bbb49ed0a440cc238b8ccb7ab WatchSource:0}: Error finding container c3eaec711be3d1a7adb1a18ccbaada3f16ebb87bbb49ed0a440cc238b8ccb7ab: Status 404 returned error can't find the container with id c3eaec711be3d1a7adb1a18ccbaada3f16ebb87bbb49ed0a440cc238b8ccb7ab
Jul 09 18:04:00.413096 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.412997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:00.413096 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.413040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:00.413096 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.413087    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:00.413377 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.413171    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:00.413377 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.413189    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5fdd7ff6-khzmb: secret "image-registry-tls" not found
Jul 09 18:04:00.413377 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.413205    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:00.413377 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.413223    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:00.413377 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.413252    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls podName:f5507b99-2a64-4f58-a456-4c80b25edf93 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:01.413233709 +0000 UTC m=+37.059227239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls") pod "image-registry-5c5fdd7ff6-khzmb" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93") : secret "image-registry-tls" not found
Jul 09 18:04:00.413680 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.413668    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:01.413261077 +0000 UTC m=+37.059254593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:00.413718 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.413689    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:01.413683757 +0000 UTC m=+37.059677274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:00.514141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:00.514104    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:00.514300 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.514258    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:00.514368 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:00.514320    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls podName:ec64976d-0e4f-4c30-9203-b000c1597408 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:01.514303053 +0000 UTC m=+37.160296584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls") pod "dns-default-b2gwz" (UID: "ec64976d-0e4f-4c30-9203-b000c1597408") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:01.110564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:01.110519    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9gflx" event={"ID":"f446a6a5-d53a-4147-a454-18dc8917eadd","Type":"ContainerStarted","Data":"c7d87e76383eba7e06b8939d6f142f8ca0d37695b7835b70723e6c48b933ee7d"}
Jul 09 18:04:01.111739 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:01.111708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2" event={"ID":"3be24e31-d32d-4306-bd83-bb3fca6b64b5","Type":"ContainerStarted","Data":"c3eaec711be3d1a7adb1a18ccbaada3f16ebb87bbb49ed0a440cc238b8ccb7ab"}
Jul 09 18:04:01.423214 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:01.423126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:01.423214 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:01.423182    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:01.423451 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:01.423224    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:01.423451 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:01.423279    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:01.423451 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:01.423299    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5fdd7ff6-khzmb: secret "image-registry-tls" not found
Jul 09 18:04:01.423451 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:01.423315    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:01.423451 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:01.423352    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:01.423451 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:01.423378    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls podName:f5507b99-2a64-4f58-a456-4c80b25edf93 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:03.423360224 +0000 UTC m=+39.069353741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls") pod "image-registry-5c5fdd7ff6-khzmb" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93") : secret "image-registry-tls" not found
Jul 09 18:04:01.423451 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:01.423397    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:03.423383877 +0000 UTC m=+39.069377393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:01.423451 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:01.423412    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:03.423405728 +0000 UTC m=+39.069399245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:01.524339 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:01.524298    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:01.524592 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:01.524569    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:01.524709 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:01.524694    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls podName:ec64976d-0e4f-4c30-9203-b000c1597408 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:03.524671187 +0000 UTC m=+39.170664720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls") pod "dns-default-b2gwz" (UID: "ec64976d-0e4f-4c30-9203-b000c1597408") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:03.441387 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:03.441318    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:03.441387 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:03.441388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:03.441914 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:03.441427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:03.441914 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:03.441528    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:03.441914 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:03.441547    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5fdd7ff6-khzmb: secret "image-registry-tls" not found
Jul 09 18:04:03.441914 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:03.441556    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:03.441914 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:03.441571    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:03.441914 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:03.441603    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls podName:f5507b99-2a64-4f58-a456-4c80b25edf93 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:07.441588003 +0000 UTC m=+43.087581524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls") pod "image-registry-5c5fdd7ff6-khzmb" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93") : secret "image-registry-tls" not found
Jul 09 18:04:03.441914 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:03.441627    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:07.441609066 +0000 UTC m=+43.087602603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:03.441914 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:03.441645    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:07.44163575 +0000 UTC m=+43.087629303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:03.542122 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:03.541873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:03.542303 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:03.542003    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:03.542303 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:03.542285    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls podName:ec64976d-0e4f-4c30-9203-b000c1597408 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:07.542257764 +0000 UTC m=+43.188251296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls") pod "dns-default-b2gwz" (UID: "ec64976d-0e4f-4c30-9203-b000c1597408") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:06.126152 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:06.126060    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2" event={"ID":"3be24e31-d32d-4306-bd83-bb3fca6b64b5","Type":"ContainerStarted","Data":"1948df6c9808aff0c7a75a4744ac3949915ebbfb28b1f1f7f23a1aca22b16594"}
Jul 09 18:04:06.128945 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:06.128920    2558 generic.go:356] "Generic (PLEG): container finished" podID="a677a1da-0454-4e1e-9842-729623211e74" containerID="96b886a70f083652858f8439f67068479fbd366977008fcaad0505ac92429c37" exitCode=0
Jul 09 18:04:06.129088 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:06.128993    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbvg5" event={"ID":"a677a1da-0454-4e1e-9842-729623211e74","Type":"ContainerDied","Data":"96b886a70f083652858f8439f67068479fbd366977008fcaad0505ac92429c37"}
Jul 09 18:04:06.130574 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:06.130540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9gflx" event={"ID":"f446a6a5-d53a-4147-a454-18dc8917eadd","Type":"ContainerStarted","Data":"72b3c3c7313cf62e690f4d7148a1f63a5fc3b62314ec94176b9811e504ceb4a7"}
Jul 09 18:04:06.149651 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:06.149590    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-b8vj2" podStartSLOduration=1.612496992 podStartE2EDuration="7.149571966s" totalImagesPullingTime="5.537074974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:06.146807238 +0000 UTC m=+41.792800781" watchObservedRunningTime="2026-07-09 18:04:06.149571966 +0000 UTC m=+41.795565507"
Jul 09 18:04:06.167582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:06.167533    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9gflx" podStartSLOduration=1.6327590220000001 podStartE2EDuration="7.167514715s" totalImagesPullingTime="5.534755693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:06.166953766 +0000 UTC m=+41.812947320" watchObservedRunningTime="2026-07-09 18:04:06.167514715 +0000 UTC m=+41.813508258"
Jul 09 18:04:07.137180 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:07.137068    2558 generic.go:356] "Generic (PLEG): container finished" podID="a677a1da-0454-4e1e-9842-729623211e74" containerID="9922609f9d5c849a23490bca5722fba2594204afa2cce107073b6e8f439dae25" exitCode=0
Jul 09 18:04:07.137641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:07.137206    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbvg5" event={"ID":"a677a1da-0454-4e1e-9842-729623211e74","Type":"ContainerDied","Data":"9922609f9d5c849a23490bca5722fba2594204afa2cce107073b6e8f439dae25"}
Jul 09 18:04:07.472380 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:07.472346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:07.472548 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:07.472387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:07.472548 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:07.472481    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:07.472548 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:07.472487    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:07.472548 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:07.472508    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c5fdd7ff6-khzmb: secret "image-registry-tls" not found
Jul 09 18:04:07.472548 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:07.472519    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:07.472548 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:07.472538    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:15.472524028 +0000 UTC m=+51.118517544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:07.472853 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:07.472554    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls podName:f5507b99-2a64-4f58-a456-4c80b25edf93 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:15.472548145 +0000 UTC m=+51.118541662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls") pod "image-registry-5c5fdd7ff6-khzmb" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93") : secret "image-registry-tls" not found
Jul 09 18:04:07.472853 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:07.472579    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:04:07.472853 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:07.472629    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:15.472613531 +0000 UTC m=+51.118607053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : configmap "kube-rbac-proxy" not found
Jul 09 18:04:07.573555 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:07.573522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:07.573744 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:07.573643    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:07.573744 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:07.573691    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls podName:ec64976d-0e4f-4c30-9203-b000c1597408 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:15.573678551 +0000 UTC m=+51.219672068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls") pod "dns-default-b2gwz" (UID: "ec64976d-0e4f-4c30-9203-b000c1597408") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:08.143768 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:08.143733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbvg5" event={"ID":"a677a1da-0454-4e1e-9842-729623211e74","Type":"ContainerStarted","Data":"5722ae133c52b6c19c90d2a6781350165ffe0de550721e91a937d3b98cc4ac87"}
Jul 09 18:04:08.171019 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:08.170973    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cbvg5" podStartSLOduration=6.66110234 podStartE2EDuration="43.17095633s" totalImagesPullingTime="36.50985399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:08.168709197 +0000 UTC m=+43.814702736" watchObservedRunningTime="2026-07-09 18:04:08.17095633 +0000 UTC m=+43.816949869"
Jul 09 18:04:15.536508 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.536461    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:15.536508 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.536509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:15.537117 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.536537    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:15.537117 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:15.536648    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:15.537117 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:15.536723    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls podName:b472c8fe-f511-4965-91fa-237ad1fd8f6d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:31.536708431 +0000 UTC m=+67.182701950 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmvpf" (UID: "b472c8fe-f511-4965-91fa-237ad1fd8f6d") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:04:15.537259 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.537233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b472c8fe-f511-4965-91fa-237ad1fd8f6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:15.540340 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.540308    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") pod \"image-registry-5c5fdd7ff6-khzmb\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") " pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:15.604530 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.604482    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:15.637621 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.637588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:15.640026 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.639997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec64976d-0e4f-4c30-9203-b000c1597408-metrics-tls\") pod \"dns-default-b2gwz\" (UID: \"ec64976d-0e4f-4c30-9203-b000c1597408\") " pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:15.662283 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.662045    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:15.793670 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.793599    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"]
Jul 09 18:04:15.797619 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:15.797593    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5507b99_2a64_4f58_a456_4c80b25edf93.slice/crio-b910249def9a05916afc402d2fd0cf1110a463982e87c0e8ba15d0a2941f38da WatchSource:0}: Error finding container b910249def9a05916afc402d2fd0cf1110a463982e87c0e8ba15d0a2941f38da: Status 404 returned error can't find the container with id b910249def9a05916afc402d2fd0cf1110a463982e87c0e8ba15d0a2941f38da
Jul 09 18:04:15.827136 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:15.827028    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b2gwz"]
Jul 09 18:04:15.829349 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:15.829309    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec64976d_0e4f_4c30_9203_b000c1597408.slice/crio-2d1390f99195d7cb1fbc1d9b76626d87ae49a2b4166e27ed4e3bbc08d1915cc6 WatchSource:0}: Error finding container 2d1390f99195d7cb1fbc1d9b76626d87ae49a2b4166e27ed4e3bbc08d1915cc6: Status 404 returned error can't find the container with id 2d1390f99195d7cb1fbc1d9b76626d87ae49a2b4166e27ed4e3bbc08d1915cc6
Jul 09 18:04:16.163054 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:16.162966    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2gwz" event={"ID":"ec64976d-0e4f-4c30-9203-b000c1597408","Type":"ContainerStarted","Data":"2d1390f99195d7cb1fbc1d9b76626d87ae49a2b4166e27ed4e3bbc08d1915cc6"}
Jul 09 18:04:16.164577 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:16.164548    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb" event={"ID":"f5507b99-2a64-4f58-a456-4c80b25edf93","Type":"ContainerStarted","Data":"4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22"}
Jul 09 18:04:16.164714 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:16.164595    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb" event={"ID":"f5507b99-2a64-4f58-a456-4c80b25edf93","Type":"ContainerStarted","Data":"b910249def9a05916afc402d2fd0cf1110a463982e87c0e8ba15d0a2941f38da"}
Jul 09 18:04:16.164757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:16.164708    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:16.200246 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:16.200186    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb" podStartSLOduration=27.200163921 podStartE2EDuration="27.200163921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:16.199804428 +0000 UTC m=+51.845797967" watchObservedRunningTime="2026-07-09 18:04:16.200163921 +0000 UTC m=+51.846157461"
Jul 09 18:04:19.173903 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:19.173869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2gwz" event={"ID":"ec64976d-0e4f-4c30-9203-b000c1597408","Type":"ContainerStarted","Data":"59b856950711763064885ff91aac5e88d0402cafb274038477649a0cd73a80af"}
Jul 09 18:04:19.174294 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:19.173920    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2gwz" event={"ID":"ec64976d-0e4f-4c30-9203-b000c1597408","Type":"ContainerStarted","Data":"a852504eb2726ff4dff3bf6596006c86be5b20b9b11f7235a3a14600f854d37c"}
Jul 09 18:04:19.174294 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:19.174005    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:19.195942 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:19.195887    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b2gwz" podStartSLOduration=17.771213461 podStartE2EDuration="20.195871316s" totalImagesPullingTime="2.424657855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:19.195316827 +0000 UTC m=+54.841310380" watchObservedRunningTime="2026-07-09 18:04:19.195871316 +0000 UTC m=+54.841864833"
Jul 09 18:04:21.574527 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.574495    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"]
Jul 09 18:04:21.576529 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.576513    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"
Jul 09 18:04:21.581910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.581889    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 18:04:21.582045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.581890    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 18:04:21.583945 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.583918    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 18:04:21.584040 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.583923    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nrs4s\""
Jul 09 18:04:21.584040 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.583928    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 18:04:21.587267 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.587246    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"]
Jul 09 18:04:21.588982 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.588969    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.592981 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.592956    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 18:04:21.593209 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.593189    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 18:04:21.593841 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.593825    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 18:04:21.593978 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.593923    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"]
Jul 09 18:04:21.594664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.594646    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 18:04:21.600378 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.600358    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"]
Jul 09 18:04:21.602016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.601999    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.605235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.605217    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 18:04:21.610811 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.610787    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"]
Jul 09 18:04:21.612589 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.612555    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"]
Jul 09 18:04:21.643369 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.643342    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"]
Jul 09 18:04:21.687401 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.687375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.687536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.687421    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/112121de-4f10-42f1-a428-b3e77b064c1b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.687536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.687456    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-hub\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.687536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.687517    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f018599-c0ba-4234-8b3a-02d37291cf9b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cb977db44-qpbnt\" (UID: \"5f018599-c0ba-4234-8b3a-02d37291cf9b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"
Jul 09 18:04:21.687653 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.687544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-ca\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.687653 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.687570    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79x2f\" (UniqueName: \"kubernetes.io/projected/5f018599-c0ba-4234-8b3a-02d37291cf9b-kube-api-access-79x2f\") pod \"managed-serviceaccount-addon-agent-5cb977db44-qpbnt\" (UID: \"5f018599-c0ba-4234-8b3a-02d37291cf9b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"
Jul 09 18:04:21.687653 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.687598    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.687653 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.687623    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgqs\" (UniqueName: \"kubernetes.io/projected/112121de-4f10-42f1-a428-b3e77b064c1b-kube-api-access-wbgqs\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.728103 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.728073    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7775456cb8-9znrq"]
Jul 09 18:04:21.730713 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.730684    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.748376 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.748351    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7775456cb8-9znrq"]
Jul 09 18:04:21.788592 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788567    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/112121de-4f10-42f1-a428-b3e77b064c1b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.788765 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-hub\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.788765 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9e9477a-b71b-4120-9476-1f8da6dac207-tmp\") pod \"klusterlet-addon-workmgr-64fd6b5b5f-gr62j\" (UID: \"b9e9477a-b71b-4120-9476-1f8da6dac207\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.788765 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788674    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f018599-c0ba-4234-8b3a-02d37291cf9b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cb977db44-qpbnt\" (UID: \"5f018599-c0ba-4234-8b3a-02d37291cf9b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"
Jul 09 18:04:21.788765 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-ca\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.788765 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79x2f\" (UniqueName: \"kubernetes.io/projected/5f018599-c0ba-4234-8b3a-02d37291cf9b-kube-api-access-79x2f\") pod \"managed-serviceaccount-addon-agent-5cb977db44-qpbnt\" (UID: \"5f018599-c0ba-4234-8b3a-02d37291cf9b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"
Jul 09 18:04:21.789026 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788769    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.789026 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgqs\" (UniqueName: \"kubernetes.io/projected/112121de-4f10-42f1-a428-b3e77b064c1b-kube-api-access-wbgqs\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.789026 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788901    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.789026 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.788937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b9e9477a-b71b-4120-9476-1f8da6dac207-klusterlet-config\") pod \"klusterlet-addon-workmgr-64fd6b5b5f-gr62j\" (UID: \"b9e9477a-b71b-4120-9476-1f8da6dac207\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.789196 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.789066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvp8g\" (UniqueName: \"kubernetes.io/projected/b9e9477a-b71b-4120-9476-1f8da6dac207-kube-api-access-vvp8g\") pod \"klusterlet-addon-workmgr-64fd6b5b5f-gr62j\" (UID: \"b9e9477a-b71b-4120-9476-1f8da6dac207\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.789444 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.789420    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/112121de-4f10-42f1-a428-b3e77b064c1b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.791631 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.791587    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.791631 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.791619    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-ca\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.791789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.791656    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-hub\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.792072 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.792049    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f018599-c0ba-4234-8b3a-02d37291cf9b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cb977db44-qpbnt\" (UID: \"5f018599-c0ba-4234-8b3a-02d37291cf9b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"
Jul 09 18:04:21.792178 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.792163    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/112121de-4f10-42f1-a428-b3e77b064c1b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.800057 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.800036    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79x2f\" (UniqueName: \"kubernetes.io/projected/5f018599-c0ba-4234-8b3a-02d37291cf9b-kube-api-access-79x2f\") pod \"managed-serviceaccount-addon-agent-5cb977db44-qpbnt\" (UID: \"5f018599-c0ba-4234-8b3a-02d37291cf9b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"
Jul 09 18:04:21.800624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.800607    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgqs\" (UniqueName: \"kubernetes.io/projected/112121de-4f10-42f1-a428-b3e77b064c1b-kube-api-access-wbgqs\") pod \"cluster-proxy-proxy-agent-647465d5d8-4fxtd\" (UID: \"112121de-4f10-42f1-a428-b3e77b064c1b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.889857 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.889773    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97da9de6-c59a-452f-b85e-8d92de447407-image-registry-private-configuration\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.889857 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.889815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97da9de6-c59a-452f-b85e-8d92de447407-registry-certificates\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.889857 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.889836    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97da9de6-c59a-452f-b85e-8d92de447407-trusted-ca\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.890066 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.889879    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5l2q\" (UniqueName: \"kubernetes.io/projected/97da9de6-c59a-452f-b85e-8d92de447407-kube-api-access-h5l2q\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.890066 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.889920    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97da9de6-c59a-452f-b85e-8d92de447407-ca-trust-extracted\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.890066 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.889938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b9e9477a-b71b-4120-9476-1f8da6dac207-klusterlet-config\") pod \"klusterlet-addon-workmgr-64fd6b5b5f-gr62j\" (UID: \"b9e9477a-b71b-4120-9476-1f8da6dac207\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.890066 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.889956    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97da9de6-c59a-452f-b85e-8d92de447407-registry-tls\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.890066 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.889979    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97da9de6-c59a-452f-b85e-8d92de447407-installation-pull-secrets\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.890066 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.889998    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvp8g\" (UniqueName: \"kubernetes.io/projected/b9e9477a-b71b-4120-9476-1f8da6dac207-kube-api-access-vvp8g\") pod \"klusterlet-addon-workmgr-64fd6b5b5f-gr62j\" (UID: \"b9e9477a-b71b-4120-9476-1f8da6dac207\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.890066 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.890027    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97da9de6-c59a-452f-b85e-8d92de447407-bound-sa-token\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.890272 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.890088    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9e9477a-b71b-4120-9476-1f8da6dac207-tmp\") pod \"klusterlet-addon-workmgr-64fd6b5b5f-gr62j\" (UID: \"b9e9477a-b71b-4120-9476-1f8da6dac207\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.890491 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.890473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9e9477a-b71b-4120-9476-1f8da6dac207-tmp\") pod \"klusterlet-addon-workmgr-64fd6b5b5f-gr62j\" (UID: \"b9e9477a-b71b-4120-9476-1f8da6dac207\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.892555 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.892534    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b9e9477a-b71b-4120-9476-1f8da6dac207-klusterlet-config\") pod \"klusterlet-addon-workmgr-64fd6b5b5f-gr62j\" (UID: \"b9e9477a-b71b-4120-9476-1f8da6dac207\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.900510 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.900488    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"
Jul 09 18:04:21.904859 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.904837    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvp8g\" (UniqueName: \"kubernetes.io/projected/b9e9477a-b71b-4120-9476-1f8da6dac207-kube-api-access-vvp8g\") pod \"klusterlet-addon-workmgr-64fd6b5b5f-gr62j\" (UID: \"b9e9477a-b71b-4120-9476-1f8da6dac207\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.907706 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.907685    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"
Jul 09 18:04:21.924948 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.924915    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.991265    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5l2q\" (UniqueName: \"kubernetes.io/projected/97da9de6-c59a-452f-b85e-8d92de447407-kube-api-access-h5l2q\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.991376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97da9de6-c59a-452f-b85e-8d92de447407-ca-trust-extracted\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.991409    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97da9de6-c59a-452f-b85e-8d92de447407-registry-tls\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.991442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97da9de6-c59a-452f-b85e-8d92de447407-installation-pull-secrets\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.991485    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97da9de6-c59a-452f-b85e-8d92de447407-bound-sa-token\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.991560    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97da9de6-c59a-452f-b85e-8d92de447407-image-registry-private-configuration\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.991597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97da9de6-c59a-452f-b85e-8d92de447407-registry-certificates\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.991620    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97da9de6-c59a-452f-b85e-8d92de447407-trusted-ca\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.992727    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97da9de6-c59a-452f-b85e-8d92de447407-ca-trust-extracted\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.992936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97da9de6-c59a-452f-b85e-8d92de447407-trusted-ca\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.993720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.993671    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97da9de6-c59a-452f-b85e-8d92de447407-registry-certificates\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.995768 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.995739    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97da9de6-c59a-452f-b85e-8d92de447407-image-registry-private-configuration\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.995876 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.995857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97da9de6-c59a-452f-b85e-8d92de447407-registry-tls\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:21.997756 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:21.997671    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97da9de6-c59a-452f-b85e-8d92de447407-installation-pull-secrets\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:22.011653 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.011628    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97da9de6-c59a-452f-b85e-8d92de447407-bound-sa-token\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:22.011941 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.011920    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5l2q\" (UniqueName: \"kubernetes.io/projected/97da9de6-c59a-452f-b85e-8d92de447407-kube-api-access-h5l2q\") pod \"image-registry-7775456cb8-9znrq\" (UID: \"97da9de6-c59a-452f-b85e-8d92de447407\") " pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:22.039674 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.039643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:22.052870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.051631    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt"]
Jul 09 18:04:22.057566 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:22.057523    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f018599_c0ba_4234_8b3a_02d37291cf9b.slice/crio-bd31d4524afbb002f8715031555a5c127894c80163f3e862d28b4dc2be00e207 WatchSource:0}: Error finding container bd31d4524afbb002f8715031555a5c127894c80163f3e862d28b4dc2be00e207: Status 404 returned error can't find the container with id bd31d4524afbb002f8715031555a5c127894c80163f3e862d28b4dc2be00e207
Jul 09 18:04:22.069127 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.069102    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd"]
Jul 09 18:04:22.072194 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:22.072170    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod112121de_4f10_42f1_a428_b3e77b064c1b.slice/crio-cda615095326e7d3c389ba0e24db9123d6961d7271bc97946c24d254e757911b WatchSource:0}: Error finding container cda615095326e7d3c389ba0e24db9123d6961d7271bc97946c24d254e757911b: Status 404 returned error can't find the container with id cda615095326e7d3c389ba0e24db9123d6961d7271bc97946c24d254e757911b
Jul 09 18:04:22.087114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.087087    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"]
Jul 09 18:04:22.089888 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:22.089865    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e9477a_b71b_4120_9476_1f8da6dac207.slice/crio-648ecf63b8fa428e16eb667828a6044f16bdea1e15eefb35562aac10684e886f WatchSource:0}: Error finding container 648ecf63b8fa428e16eb667828a6044f16bdea1e15eefb35562aac10684e886f: Status 404 returned error can't find the container with id 648ecf63b8fa428e16eb667828a6044f16bdea1e15eefb35562aac10684e886f
Jul 09 18:04:22.176654 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.176579    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7775456cb8-9znrq"]
Jul 09 18:04:22.179873 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:22.179846    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97da9de6_c59a_452f_b85e_8d92de447407.slice/crio-9f9b6bd8d04b46d3485f2d0d547d4e4d371216a9048ce6aed17cd9c38e7d3af2 WatchSource:0}: Error finding container 9f9b6bd8d04b46d3485f2d0d547d4e4d371216a9048ce6aed17cd9c38e7d3af2: Status 404 returned error can't find the container with id 9f9b6bd8d04b46d3485f2d0d547d4e4d371216a9048ce6aed17cd9c38e7d3af2
Jul 09 18:04:22.183380 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.183355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd" event={"ID":"112121de-4f10-42f1-a428-b3e77b064c1b","Type":"ContainerStarted","Data":"cda615095326e7d3c389ba0e24db9123d6961d7271bc97946c24d254e757911b"}
Jul 09 18:04:22.184455 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.184436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt" event={"ID":"5f018599-c0ba-4234-8b3a-02d37291cf9b","Type":"ContainerStarted","Data":"bd31d4524afbb002f8715031555a5c127894c80163f3e862d28b4dc2be00e207"}
Jul 09 18:04:22.185498 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:22.185478    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j" event={"ID":"b9e9477a-b71b-4120-9476-1f8da6dac207","Type":"ContainerStarted","Data":"648ecf63b8fa428e16eb667828a6044f16bdea1e15eefb35562aac10684e886f"}
Jul 09 18:04:23.193453 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:23.193414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7775456cb8-9znrq" event={"ID":"97da9de6-c59a-452f-b85e-8d92de447407","Type":"ContainerStarted","Data":"4076b783d1a99da849049440fa224393570698544429d1215ecfad25e37fb574"}
Jul 09 18:04:23.193917 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:23.193512    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7775456cb8-9znrq" event={"ID":"97da9de6-c59a-452f-b85e-8d92de447407","Type":"ContainerStarted","Data":"9f9b6bd8d04b46d3485f2d0d547d4e4d371216a9048ce6aed17cd9c38e7d3af2"}
Jul 09 18:04:23.194257 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:23.194236    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:27.107941 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:27.107904    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5s8lz"
Jul 09 18:04:27.143805 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:27.143756    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7775456cb8-9znrq" podStartSLOduration=6.143740927 podStartE2EDuration="6.143740927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:23.229907353 +0000 UTC m=+58.875900894" watchObservedRunningTime="2026-07-09 18:04:27.143740927 +0000 UTC m=+62.789734466"
Jul 09 18:04:29.178764 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.178729    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b2gwz"
Jul 09 18:04:29.219317 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.219262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd" event={"ID":"112121de-4f10-42f1-a428-b3e77b064c1b","Type":"ContainerStarted","Data":"84f918cd90465c16b87e654b4dd0066fd54b4dd709c2ae19cf13da137ef9adee"}
Jul 09 18:04:29.219494 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.219356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd" event={"ID":"112121de-4f10-42f1-a428-b3e77b064c1b","Type":"ContainerStarted","Data":"3b2bad7861f15d206417af6e5c557f52c6f51023e424250ef263b56788314b93"}
Jul 09 18:04:29.219494 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.219378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd" event={"ID":"112121de-4f10-42f1-a428-b3e77b064c1b","Type":"ContainerStarted","Data":"5eddd4a87ea80d33e161cc97f15b4c5ecbfec665e53b145eef96570a72084e3e"}
Jul 09 18:04:29.220941 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.220915    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt" event={"ID":"5f018599-c0ba-4234-8b3a-02d37291cf9b","Type":"ContainerStarted","Data":"bd62b72b848018a95d628839627a42ea8ec0065d71188eb237ec5f4b540c1808"}
Jul 09 18:04:29.222316 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.222294    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j" event={"ID":"b9e9477a-b71b-4120-9476-1f8da6dac207","Type":"ContainerStarted","Data":"53fff4fd10bafd54f14d7cd010141e0684ad990497f1319b38296aee39f56e27"}
Jul 09 18:04:29.222520 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.222505    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:29.224462 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.224439    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j"
Jul 09 18:04:29.241717 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.241666    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647465d5d8-4fxtd" podStartSLOduration=2.042588804 podStartE2EDuration="8.241646272s" totalImagesPullingTime="6.199057468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.241456728 +0000 UTC m=+64.887450270" watchObservedRunningTime="2026-07-09 18:04:29.241646272 +0000 UTC m=+64.887639792"
Jul 09 18:04:29.258827 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.258786    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cb977db44-qpbnt" podStartSLOduration=2.060353475 podStartE2EDuration="8.258770433s" totalImagesPullingTime="6.198416958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.258433475 +0000 UTC m=+64.904427027" watchObservedRunningTime="2026-07-09 18:04:29.258770433 +0000 UTC m=+64.904763972"
Jul 09 18:04:29.282814 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.282769    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fd6b5b5f-gr62j" podStartSLOduration=2.103936981 podStartE2EDuration="8.282753776s" totalImagesPullingTime="6.178816795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.280127943 +0000 UTC m=+64.926121482" watchObservedRunningTime="2026-07-09 18:04:29.282753776 +0000 UTC m=+64.928747314"
Jul 09 18:04:29.559782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.559750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:04:29.559942 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.559805    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:04:29.563179 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.563160    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:04:29.563179 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.563172    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:04:29.572512 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.572485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7fde25b-60ca-405a-acf7-53e67418af9f-metrics-certs\") pod \"network-metrics-daemon-5kshf\" (UID: \"c7fde25b-60ca-405a-acf7-53e67418af9f\") " pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:04:29.572614 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.572517    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/283ad84b-60d3-43ab-80e6-33da3e63bc77-original-pull-secret\") pod \"global-pull-secret-syncer-ql4td\" (UID: \"283ad84b-60d3-43ab-80e6-33da3e63bc77\") " pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:04:29.602724 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.602692    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ql4td"
Jul 09 18:04:29.610897 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.610875    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cmkm4\""
Jul 09 18:04:29.618007 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.617986    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kshf"
Jul 09 18:04:29.661257 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.660788    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:04:29.664775 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.664743    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:04:29.675498 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.675348    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:04:29.684421 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.684376    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzdg\" (UniqueName: \"kubernetes.io/projected/60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00-kube-api-access-hbzdg\") pod \"network-check-target-tsz48\" (UID: \"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00\") " pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:04:29.738832 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.738801    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ql4td"]
Jul 09 18:04:29.741489 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:29.741457    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283ad84b_60d3_43ab_80e6_33da3e63bc77.slice/crio-d3cc600e4da56b1178c06c350d54d0e44823d6db675f670a1224d66c322c5f46 WatchSource:0}: Error finding container d3cc600e4da56b1178c06c350d54d0e44823d6db675f670a1224d66c322c5f46: Status 404 returned error can't find the container with id d3cc600e4da56b1178c06c350d54d0e44823d6db675f670a1224d66c322c5f46
Jul 09 18:04:29.755547 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.755509    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kshf"]
Jul 09 18:04:29.758661 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:29.758640    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fde25b_60ca_405a_acf7_53e67418af9f.slice/crio-e52ef24990ca846b58aa2b563de41ea2858cf679724d40b70adf6a7de5f06eb5 WatchSource:0}: Error finding container e52ef24990ca846b58aa2b563de41ea2858cf679724d40b70adf6a7de5f06eb5: Status 404 returned error can't find the container with id e52ef24990ca846b58aa2b563de41ea2858cf679724d40b70adf6a7de5f06eb5
Jul 09 18:04:29.901190 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.901115    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hzqhq\""
Jul 09 18:04:29.908095 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:29.908075    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:04:30.031174 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:30.031149    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tsz48"]
Jul 09 18:04:30.033380 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:30.033350    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e3b7f5_5ea8_40a9_a4f5_6bf7adfeca00.slice/crio-1567908b6811001a91bbbeebff9c26c106ff854e5cc7500436cb9a7cadc20f3c WatchSource:0}: Error finding container 1567908b6811001a91bbbeebff9c26c106ff854e5cc7500436cb9a7cadc20f3c: Status 404 returned error can't find the container with id 1567908b6811001a91bbbeebff9c26c106ff854e5cc7500436cb9a7cadc20f3c
Jul 09 18:04:30.225895 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:30.225862    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kshf" event={"ID":"c7fde25b-60ca-405a-acf7-53e67418af9f","Type":"ContainerStarted","Data":"e52ef24990ca846b58aa2b563de41ea2858cf679724d40b70adf6a7de5f06eb5"}
Jul 09 18:04:30.226948 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:30.226918    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tsz48" event={"ID":"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00","Type":"ContainerStarted","Data":"1567908b6811001a91bbbeebff9c26c106ff854e5cc7500436cb9a7cadc20f3c"}
Jul 09 18:04:30.228124 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:30.228103    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ql4td" event={"ID":"283ad84b-60d3-43ab-80e6-33da3e63bc77","Type":"ContainerStarted","Data":"d3cc600e4da56b1178c06c350d54d0e44823d6db675f670a1224d66c322c5f46"}
Jul 09 18:04:31.237650 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:31.237567    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kshf" event={"ID":"c7fde25b-60ca-405a-acf7-53e67418af9f","Type":"ContainerStarted","Data":"2d47e25c40aae7242b6d8a69cd85dfe9fca0380df5536370001eeb795bfeef40"}
Jul 09 18:04:31.237967 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:31.237668    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kshf" event={"ID":"c7fde25b-60ca-405a-acf7-53e67418af9f","Type":"ContainerStarted","Data":"0dcc69c6355738e898b46009c1d989330894e4644a77d852109103f945030fff"}
Jul 09 18:04:31.257707 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:31.257657    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5kshf" podStartSLOduration=65.068959029 podStartE2EDuration="1m6.257643032s" totalImagesPullingTime="1.188684003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:31.256622818 +0000 UTC m=+66.902616357" watchObservedRunningTime="2026-07-09 18:04:31.257643032 +0000 UTC m=+66.903636572"
Jul 09 18:04:31.577434 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:31.577353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:31.580254 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:31.580230    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b472c8fe-f511-4965-91fa-237ad1fd8f6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmvpf\" (UID: \"b472c8fe-f511-4965-91fa-237ad1fd8f6d\") " pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:31.650859 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:31.650829    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:31.837545 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:31.837461    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f5z8w\""
Jul 09 18:04:31.844363 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:31.844313    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmvpf"
Jul 09 18:04:31.991678 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:31.991598    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmvpf"]
Jul 09 18:04:31.997274 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:31.997240    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb472c8fe_f511_4965_91fa_237ad1fd8f6d.slice/crio-f2b555a1e7f8e440f3fc8d1fc09200f1b7b943842f11e91d005a85c201e63efb WatchSource:0}: Error finding container f2b555a1e7f8e440f3fc8d1fc09200f1b7b943842f11e91d005a85c201e63efb: Status 404 returned error can't find the container with id f2b555a1e7f8e440f3fc8d1fc09200f1b7b943842f11e91d005a85c201e63efb
Jul 09 18:04:32.242088 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:32.242031    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmvpf" event={"ID":"b472c8fe-f511-4965-91fa-237ad1fd8f6d","Type":"ContainerStarted","Data":"f2b555a1e7f8e440f3fc8d1fc09200f1b7b943842f11e91d005a85c201e63efb"}
Jul 09 18:04:36.509348 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.509301    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5bjwn"]
Jul 09 18:04:36.594756 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.594723    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.599575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.599212    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:04:36.599575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.599289    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:04:36.599575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.599220    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:04:36.599575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.599213    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:04:36.600994 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.600871    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:04:36.600994 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.600897    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:04:36.601168 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.601106    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ztzgw\""
Jul 09 18:04:36.725523 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.725493    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e9235fc-c082-4977-b7e9-a8d08b751910-root\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.725757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.725540    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-textfile\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.725757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.725578    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-tls\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.725757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.725604    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4js2\" (UniqueName: \"kubernetes.io/projected/8e9235fc-c082-4977-b7e9-a8d08b751910-kube-api-access-h4js2\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.725757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.725670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e9235fc-c082-4977-b7e9-a8d08b751910-sys\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.725757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.725712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.725757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.725751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e9235fc-c082-4977-b7e9-a8d08b751910-metrics-client-ca\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.726053 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.725776    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-wtmp\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.726053 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.725819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.826604 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826530    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.826758 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826603    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e9235fc-c082-4977-b7e9-a8d08b751910-root\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.826758 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826630    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-textfile\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.826758 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-tls\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.826758 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4js2\" (UniqueName: \"kubernetes.io/projected/8e9235fc-c082-4977-b7e9-a8d08b751910-kube-api-access-h4js2\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.826758 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e9235fc-c082-4977-b7e9-a8d08b751910-sys\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.826758 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826737    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.827031 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826775    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e9235fc-c082-4977-b7e9-a8d08b751910-metrics-client-ca\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.827031 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-wtmp\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.827031 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.826987    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-wtmp\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.827170 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.827044    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e9235fc-c082-4977-b7e9-a8d08b751910-root\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.827170 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.827109    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-accelerators-collector-config\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.827274 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.827172    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e9235fc-c082-4977-b7e9-a8d08b751910-sys\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.827274 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:36.827259    2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 18:04:36.827407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.827293    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-textfile\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.827407 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:36.827308    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-tls podName:8e9235fc-c082-4977-b7e9-a8d08b751910 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:37.327293267 +0000 UTC m=+72.973286800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-tls") pod "node-exporter-5bjwn" (UID: "8e9235fc-c082-4977-b7e9-a8d08b751910") : secret "node-exporter-tls" not found
Jul 09 18:04:36.827825 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.827796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e9235fc-c082-4977-b7e9-a8d08b751910-metrics-client-ca\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.829876 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.829857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:36.843534 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:36.843514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4js2\" (UniqueName: \"kubernetes.io/projected/8e9235fc-c082-4977-b7e9-a8d08b751910-kube-api-access-h4js2\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:37.332692 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:37.332659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-tls\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:37.335466 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:37.335433    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e9235fc-c082-4977-b7e9-a8d08b751910-node-exporter-tls\") pod \"node-exporter-5bjwn\" (UID: \"8e9235fc-c082-4977-b7e9-a8d08b751910\") " pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:37.507126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:37.507093    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5bjwn"
Jul 09 18:04:38.535316 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:04:38.535276    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9235fc_c082_4977_b7e9_a8d08b751910.slice/crio-28a7a00f4598ba776f39866ce00e2681d6e34db9701fd7fb0ef1a2cc176a6867 WatchSource:0}: Error finding container 28a7a00f4598ba776f39866ce00e2681d6e34db9701fd7fb0ef1a2cc176a6867: Status 404 returned error can't find the container with id 28a7a00f4598ba776f39866ce00e2681d6e34db9701fd7fb0ef1a2cc176a6867
Jul 09 18:04:39.274942 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:39.274477    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ql4td" event={"ID":"283ad84b-60d3-43ab-80e6-33da3e63bc77","Type":"ContainerStarted","Data":"74b09e9dbf419959d1023b0fb1ccc8639c82572a91adbebe21535287ebbdb7a2"}
Jul 09 18:04:39.275828 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:39.275799    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bjwn" event={"ID":"8e9235fc-c082-4977-b7e9-a8d08b751910","Type":"ContainerStarted","Data":"28a7a00f4598ba776f39866ce00e2681d6e34db9701fd7fb0ef1a2cc176a6867"}
Jul 09 18:04:39.277574 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:39.277547    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmvpf" event={"ID":"b472c8fe-f511-4965-91fa-237ad1fd8f6d","Type":"ContainerStarted","Data":"5c05a8745d6f57cfea9522f75a96dcbe6563577b702dc3b394bc3b1b5eba9bd8"}
Jul 09 18:04:39.279030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:39.279003    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tsz48" event={"ID":"60e3b7f5-5ea8-40a9-a4f5-6bf7adfeca00","Type":"ContainerStarted","Data":"4fc4c21a718ba25e76ab5951841a4b5d991f02269098cdc033a03a4599d9ff92"}
Jul 09 18:04:39.279158 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:39.279144    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:04:39.321279 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:39.321226    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ql4td" podStartSLOduration=65.53216713 podStartE2EDuration="1m14.32120888s" totalImagesPullingTime="8.78904175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:39.298513252 +0000 UTC m=+74.944506791" watchObservedRunningTime="2026-07-09 18:04:39.32120888 +0000 UTC m=+74.967202429"
Jul 09 18:04:39.321993 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:39.321959    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tsz48" podStartSLOduration=65.825986171 podStartE2EDuration="1m14.321948646s" totalImagesPullingTime="8.495962475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:39.321197917 +0000 UTC m=+74.967191454" watchObservedRunningTime="2026-07-09 18:04:39.321948646 +0000 UTC m=+74.967942234"
Jul 09 18:04:40.284612 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:40.284574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmvpf" event={"ID":"b472c8fe-f511-4965-91fa-237ad1fd8f6d","Type":"ContainerStarted","Data":"7b795e2559376a3128494fd871cca49953900ca754e95aee6a5705c260eb301b"}
Jul 09 18:04:40.286235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:40.286209    2558 generic.go:356] "Generic (PLEG): container finished" podID="8e9235fc-c082-4977-b7e9-a8d08b751910" containerID="db3955809453e0552c6e4213df66d06e45fa17e9e14a0c81e2b3d65379781585" exitCode=0
Jul 09 18:04:40.286393 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:40.286300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bjwn" event={"ID":"8e9235fc-c082-4977-b7e9-a8d08b751910","Type":"ContainerDied","Data":"db3955809453e0552c6e4213df66d06e45fa17e9e14a0c81e2b3d65379781585"}
Jul 09 18:04:41.293821 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:41.293772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bjwn" event={"ID":"8e9235fc-c082-4977-b7e9-a8d08b751910","Type":"ContainerStarted","Data":"7495cfadba08a203c175b7b57159c7a1ddd70aed528a5656164e8250c1e5e7f8"}
Jul 09 18:04:41.294261 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:41.293838    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5bjwn" event={"ID":"8e9235fc-c082-4977-b7e9-a8d08b751910","Type":"ContainerStarted","Data":"28ac45f9bf7518604331547fcc3cdeadbe8013eca89008f4619572d17a587590"}
Jul 09 18:04:41.370346 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:41.370266    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5bjwn" podStartSLOduration=4.21128281 podStartE2EDuration="5.370245034s" totalImagesPullingTime="1.0840325s" totalInitContainerRuntime="74.929724ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:41.369995601 +0000 UTC m=+77.015989136" watchObservedRunningTime="2026-07-09 18:04:41.370245034 +0000 UTC m=+77.016238575"
Jul 09 18:04:42.298838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:42.298804    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmvpf" event={"ID":"b472c8fe-f511-4965-91fa-237ad1fd8f6d","Type":"ContainerStarted","Data":"0459882f74266bb294c3a3f36a016c6e49227b4674303c779156b08c047ef7c2"}
Jul 09 18:04:42.329270 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:42.329229    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kmvpf" podStartSLOduration=33.272838512 podStartE2EDuration="43.329214101s" totalImagesPullingTime="10.056375589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:42.328969697 +0000 UTC m=+77.974963232" watchObservedRunningTime="2026-07-09 18:04:42.329214101 +0000 UTC m=+77.975207644"
Jul 09 18:04:45.207759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:45.207727    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7775456cb8-9znrq"
Jul 09 18:04:46.663191 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:46.663150    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb" podUID="f5507b99-2a64-4f58-a456-4c80b25edf93" containerName="registry" containerID="cri-o://4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22" gracePeriod=30
Jul 09 18:04:46.913037 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:46.913016    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:47.006689 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.006661    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-image-registry-private-configuration\") pod \"f5507b99-2a64-4f58-a456-4c80b25edf93\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") "
Jul 09 18:04:47.006910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.006717    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-certificates\") pod \"f5507b99-2a64-4f58-a456-4c80b25edf93\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") "
Jul 09 18:04:47.006910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.006739    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") pod \"f5507b99-2a64-4f58-a456-4c80b25edf93\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") "
Jul 09 18:04:47.006910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.006758    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4zlg\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-kube-api-access-w4zlg\") pod \"f5507b99-2a64-4f58-a456-4c80b25edf93\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") "
Jul 09 18:04:47.006910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.006799    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5507b99-2a64-4f58-a456-4c80b25edf93-ca-trust-extracted\") pod \"f5507b99-2a64-4f58-a456-4c80b25edf93\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") "
Jul 09 18:04:47.006910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.006824    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-installation-pull-secrets\") pod \"f5507b99-2a64-4f58-a456-4c80b25edf93\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") "
Jul 09 18:04:47.006910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.006855    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-trusted-ca\") pod \"f5507b99-2a64-4f58-a456-4c80b25edf93\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") "
Jul 09 18:04:47.006910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.006890    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-bound-sa-token\") pod \"f5507b99-2a64-4f58-a456-4c80b25edf93\" (UID: \"f5507b99-2a64-4f58-a456-4c80b25edf93\") "
Jul 09 18:04:47.007248 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.007195    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f5507b99-2a64-4f58-a456-4c80b25edf93" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:04:47.007858 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.007823    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f5507b99-2a64-4f58-a456-4c80b25edf93" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:04:47.009468 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.009442    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-kube-api-access-w4zlg" (OuterVolumeSpecName: "kube-api-access-w4zlg") pod "f5507b99-2a64-4f58-a456-4c80b25edf93" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93"). InnerVolumeSpecName "kube-api-access-w4zlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:04:47.009575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.009545    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f5507b99-2a64-4f58-a456-4c80b25edf93" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:04:47.009728 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.009712    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f5507b99-2a64-4f58-a456-4c80b25edf93" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:04:47.009819 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.009792    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f5507b99-2a64-4f58-a456-4c80b25edf93" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:04:47.009921 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.009904    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f5507b99-2a64-4f58-a456-4c80b25edf93" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:04:47.015877 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.015847    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5507b99-2a64-4f58-a456-4c80b25edf93-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f5507b99-2a64-4f58-a456-4c80b25edf93" (UID: "f5507b99-2a64-4f58-a456-4c80b25edf93"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:04:47.107509 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.107478    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5507b99-2a64-4f58-a456-4c80b25edf93-ca-trust-extracted\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:04:47.107509 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.107504    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-installation-pull-secrets\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:04:47.107664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.107519    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-trusted-ca\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:04:47.107664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.107530    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-bound-sa-token\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:04:47.107664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.107543    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5507b99-2a64-4f58-a456-4c80b25edf93-image-registry-private-configuration\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:04:47.107664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.107555    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-certificates\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:04:47.107664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.107567    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-registry-tls\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:04:47.107664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.107578    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4zlg\" (UniqueName: \"kubernetes.io/projected/f5507b99-2a64-4f58-a456-4c80b25edf93-kube-api-access-w4zlg\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:04:47.317307 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.317222    2558 generic.go:356] "Generic (PLEG): container finished" podID="f5507b99-2a64-4f58-a456-4c80b25edf93" containerID="4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22" exitCode=0
Jul 09 18:04:47.317307 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.317267    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb" event={"ID":"f5507b99-2a64-4f58-a456-4c80b25edf93","Type":"ContainerDied","Data":"4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22"}
Jul 09 18:04:47.317307 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.317283    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"
Jul 09 18:04:47.317307 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.317308    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5fdd7ff6-khzmb" event={"ID":"f5507b99-2a64-4f58-a456-4c80b25edf93","Type":"ContainerDied","Data":"b910249def9a05916afc402d2fd0cf1110a463982e87c0e8ba15d0a2941f38da"}
Jul 09 18:04:47.317607 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.317345    2558 scope.go:117] "RemoveContainer" containerID="4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22"
Jul 09 18:04:47.326868 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.326844    2558 scope.go:117] "RemoveContainer" containerID="4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22"
Jul 09 18:04:47.327172 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:04:47.327152    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22\": container with ID starting with 4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22 not found: ID does not exist" containerID="4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22"
Jul 09 18:04:47.327297 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.327175    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22"} err="failed to get container status \"4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22\": rpc error: code = NotFound desc = could not find container \"4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22\": container with ID starting with 4d1d025d6e1ceede8d8f0564a530124ee5ca25b29948c2ae91e7ee2a026b8b22 not found: ID does not exist"
Jul 09 18:04:47.344697 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.344668    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"]
Jul 09 18:04:47.352347 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:47.352313    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c5fdd7ff6-khzmb"]
Jul 09 18:04:48.892227 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:04:48.892192    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5507b99-2a64-4f58-a456-4c80b25edf93" path="/var/lib/kubelet/pods/f5507b99-2a64-4f58-a456-4c80b25edf93/volumes"
Jul 09 18:05:10.288814 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:05:10.288784    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tsz48"
Jul 09 18:05:16.827467 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:05:16.827435    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9gflx_f446a6a5-d53a-4147-a454-18dc8917eadd/serve-healthcheck-canary/0.log"
Jul 09 18:08:24.743567 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:08:24.743539    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:08:24.747939 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:08:24.747916    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:08:24.749516 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:08:24.749496    2558 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:11:06.330398 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.330305    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"]
Jul 09 18:11:06.330838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.330592    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5507b99-2a64-4f58-a456-4c80b25edf93" containerName="registry"
Jul 09 18:11:06.330838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.330604    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5507b99-2a64-4f58-a456-4c80b25edf93" containerName="registry"
Jul 09 18:11:06.330838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.330641    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5507b99-2a64-4f58-a456-4c80b25edf93" containerName="registry"
Jul 09 18:11:06.332387 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.332371    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"
Jul 09 18:11:06.337061 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.337041    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 09 18:11:06.337358 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.337317    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:11:06.337484 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.337432    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ggdqk\""
Jul 09 18:11:06.361831 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.361809    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"]
Jul 09 18:11:06.474033 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.474005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpdl6\" (UniqueName: \"kubernetes.io/projected/0a5ac464-5187-4b5e-883a-4abec090f814-kube-api-access-xpdl6\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h8xpr\" (UID: \"0a5ac464-5187-4b5e-883a-4abec090f814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"
Jul 09 18:11:06.474188 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.474048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a5ac464-5187-4b5e-883a-4abec090f814-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h8xpr\" (UID: \"0a5ac464-5187-4b5e-883a-4abec090f814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"
Jul 09 18:11:06.575304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.575274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpdl6\" (UniqueName: \"kubernetes.io/projected/0a5ac464-5187-4b5e-883a-4abec090f814-kube-api-access-xpdl6\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h8xpr\" (UID: \"0a5ac464-5187-4b5e-883a-4abec090f814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"
Jul 09 18:11:06.575475 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.575313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a5ac464-5187-4b5e-883a-4abec090f814-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h8xpr\" (UID: \"0a5ac464-5187-4b5e-883a-4abec090f814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"
Jul 09 18:11:06.575640 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.575625    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a5ac464-5187-4b5e-883a-4abec090f814-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h8xpr\" (UID: \"0a5ac464-5187-4b5e-883a-4abec090f814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"
Jul 09 18:11:06.590410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.590357    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpdl6\" (UniqueName: \"kubernetes.io/projected/0a5ac464-5187-4b5e-883a-4abec090f814-kube-api-access-xpdl6\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h8xpr\" (UID: \"0a5ac464-5187-4b5e-883a-4abec090f814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"
Jul 09 18:11:06.641254 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.641213    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"
Jul 09 18:11:06.786024 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.786001    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr"]
Jul 09 18:11:06.788739 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:11:06.788712    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5ac464_5187_4b5e_883a_4abec090f814.slice/crio-d1407c19b6eb31352e7f8e07b19a5add787b3b4a042664d100702603098221a7 WatchSource:0}: Error finding container d1407c19b6eb31352e7f8e07b19a5add787b3b4a042664d100702603098221a7: Status 404 returned error can't find the container with id d1407c19b6eb31352e7f8e07b19a5add787b3b4a042664d100702603098221a7
Jul 09 18:11:06.791394 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:06.791376    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:11:07.560665 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:07.560592    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr" event={"ID":"0a5ac464-5187-4b5e-883a-4abec090f814","Type":"ContainerStarted","Data":"d1407c19b6eb31352e7f8e07b19a5add787b3b4a042664d100702603098221a7"}
Jul 09 18:11:09.570369 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:09.570310    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr" event={"ID":"0a5ac464-5187-4b5e-883a-4abec090f814","Type":"ContainerStarted","Data":"91d57b5cd4908666211e588757f3f155c715b010fe25ba3cda0b4a687f332eb2"}
Jul 09 18:11:09.638257 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:09.638205    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h8xpr" podStartSLOduration=1.254896206 podStartE2EDuration="3.638190748s" totalImagesPullingTime="2.383294542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:09.637747105 +0000 UTC m=+465.283740648" watchObservedRunningTime="2026-07-09 18:11:09.638190748 +0000 UTC m=+465.284184278"
Jul 09 18:11:26.861798 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:26.861761    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"]
Jul 09 18:11:26.863866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:26.863849    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"
Jul 09 18:11:26.867871 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:26.867834    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gpkt7\""
Jul 09 18:11:26.868013 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:26.867884    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:11:26.868092 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:26.868079    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 18:11:26.880022 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:26.880002    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"]
Jul 09 18:11:27.033011 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:27.032967    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5563bfaf-42e8-4827-b6af-c9571c348468-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rdqn6\" (UID: \"5563bfaf-42e8-4827-b6af-c9571c348468\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"
Jul 09 18:11:27.033011 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:27.033009    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjsdk\" (UniqueName: \"kubernetes.io/projected/5563bfaf-42e8-4827-b6af-c9571c348468-kube-api-access-pjsdk\") pod \"openshift-lws-operator-fd8ccff4c-rdqn6\" (UID: \"5563bfaf-42e8-4827-b6af-c9571c348468\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"
Jul 09 18:11:27.133766 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:27.133671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5563bfaf-42e8-4827-b6af-c9571c348468-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rdqn6\" (UID: \"5563bfaf-42e8-4827-b6af-c9571c348468\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"
Jul 09 18:11:27.133766 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:27.133714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjsdk\" (UniqueName: \"kubernetes.io/projected/5563bfaf-42e8-4827-b6af-c9571c348468-kube-api-access-pjsdk\") pod \"openshift-lws-operator-fd8ccff4c-rdqn6\" (UID: \"5563bfaf-42e8-4827-b6af-c9571c348468\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"
Jul 09 18:11:27.134030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:27.134011    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5563bfaf-42e8-4827-b6af-c9571c348468-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rdqn6\" (UID: \"5563bfaf-42e8-4827-b6af-c9571c348468\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"
Jul 09 18:11:27.146766 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:27.146743    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjsdk\" (UniqueName: \"kubernetes.io/projected/5563bfaf-42e8-4827-b6af-c9571c348468-kube-api-access-pjsdk\") pod \"openshift-lws-operator-fd8ccff4c-rdqn6\" (UID: \"5563bfaf-42e8-4827-b6af-c9571c348468\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"
Jul 09 18:11:27.172567 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:27.172543    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"
Jul 09 18:11:27.308792 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:27.308763    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6"]
Jul 09 18:11:27.312083 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:11:27.312054    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5563bfaf_42e8_4827_b6af_c9571c348468.slice/crio-0d85dcf4c86e6ee69dfe5d9e964af057213806e08ace4985ea3841ce7c3da0c1 WatchSource:0}: Error finding container 0d85dcf4c86e6ee69dfe5d9e964af057213806e08ace4985ea3841ce7c3da0c1: Status 404 returned error can't find the container with id 0d85dcf4c86e6ee69dfe5d9e964af057213806e08ace4985ea3841ce7c3da0c1
Jul 09 18:11:27.633148 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:27.633099    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6" event={"ID":"5563bfaf-42e8-4827-b6af-c9571c348468","Type":"ContainerStarted","Data":"0d85dcf4c86e6ee69dfe5d9e964af057213806e08ace4985ea3841ce7c3da0c1"}
Jul 09 18:11:30.646642 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:30.646596    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6" event={"ID":"5563bfaf-42e8-4827-b6af-c9571c348468","Type":"ContainerStarted","Data":"aa35de4259ceb382c87614cff0a2c0a3b93b2eaf91074f501dd87f97a9c4aa89"}
Jul 09 18:11:30.673039 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:30.672990    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rdqn6" podStartSLOduration=2.205304904 podStartE2EDuration="4.672976713s" totalImagesPullingTime="2.467671809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:30.671107319 +0000 UTC m=+486.317100850" watchObservedRunningTime="2026-07-09 18:11:30.672976713 +0000 UTC m=+486.318970252"
Jul 09 18:11:52.427299 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.427267    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"]
Jul 09 18:11:52.430269 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.430249    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.434452 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.434429    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 18:11:52.436222 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.436195    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 18:11:52.436222 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.436206    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 18:11:52.436443 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.436313    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j5mvz\""
Jul 09 18:11:52.445750 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.445725    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"]
Jul 09 18:11:52.500028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.499989    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk255\" (UniqueName: \"kubernetes.io/projected/68ca30f0-e90d-4f03-88fb-d68842722870-kube-api-access-sk255\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.500028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.500035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/68ca30f0-e90d-4f03-88fb-d68842722870-manager-config\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.500274 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.500092    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/68ca30f0-e90d-4f03-88fb-d68842722870-metrics-cert\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.500274 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.500228    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ca30f0-e90d-4f03-88fb-d68842722870-cert\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.601602 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.601560    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/68ca30f0-e90d-4f03-88fb-d68842722870-metrics-cert\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.601801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.601615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ca30f0-e90d-4f03-88fb-d68842722870-cert\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.601801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.601652    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk255\" (UniqueName: \"kubernetes.io/projected/68ca30f0-e90d-4f03-88fb-d68842722870-kube-api-access-sk255\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.601801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.601692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/68ca30f0-e90d-4f03-88fb-d68842722870-manager-config\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.602373 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.602318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/68ca30f0-e90d-4f03-88fb-d68842722870-manager-config\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.604205 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.604181    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/68ca30f0-e90d-4f03-88fb-d68842722870-metrics-cert\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.604276 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.604232    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68ca30f0-e90d-4f03-88fb-d68842722870-cert\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.626908 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.626875    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk255\" (UniqueName: \"kubernetes.io/projected/68ca30f0-e90d-4f03-88fb-d68842722870-kube-api-access-sk255\") pod \"lws-controller-manager-7885457d66-lwd5g\" (UID: \"68ca30f0-e90d-4f03-88fb-d68842722870\") " pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.739237 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.739198    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:52.879696 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:52.879674    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"]
Jul 09 18:11:52.882222 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:11:52.882191    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ca30f0_e90d_4f03_88fb_d68842722870.slice/crio-1fc9f44fcf50511730287a669c097157b559254645796e36b471d243e1d499f4 WatchSource:0}: Error finding container 1fc9f44fcf50511730287a669c097157b559254645796e36b471d243e1d499f4: Status 404 returned error can't find the container with id 1fc9f44fcf50511730287a669c097157b559254645796e36b471d243e1d499f4
Jul 09 18:11:53.727371 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:53.727308    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g" event={"ID":"68ca30f0-e90d-4f03-88fb-d68842722870","Type":"ContainerStarted","Data":"1fc9f44fcf50511730287a669c097157b559254645796e36b471d243e1d499f4"}
Jul 09 18:11:54.732526 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:54.732433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g" event={"ID":"68ca30f0-e90d-4f03-88fb-d68842722870","Type":"ContainerStarted","Data":"ed5bb8eefe45944ce0d243d17a866f09c1e407195e97c8fee579af531a2ef513"}
Jul 09 18:11:54.732908 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:54.732538    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:11:54.753612 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:11:54.753564    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g" podStartSLOduration=1.176925163 podStartE2EDuration="2.75354813s" totalImagesPullingTime="1.576622967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:54.753200888 +0000 UTC m=+510.399194427" watchObservedRunningTime="2026-07-09 18:11:54.75354813 +0000 UTC m=+510.399541669"
Jul 09 18:12:05.738481 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:12:05.738453    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7885457d66-lwd5g"
Jul 09 18:13:04.814594 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:04.814557    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"]
Jul 09 18:13:04.817619 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:04.817603    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:04.821213 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:04.821195    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 18:13:04.821341 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:04.821300    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6btkn\""
Jul 09 18:13:04.821857 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:04.821843    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 18:13:04.833184 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:04.833163    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"]
Jul 09 18:13:04.947227 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:04.947199    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f848c811-6e04-4155-a455-063e0d63d7d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk\" (UID: \"f848c811-6e04-4155-a455-063e0d63d7d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:04.947401 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:04.947239    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw6mb\" (UniqueName: \"kubernetes.io/projected/f848c811-6e04-4155-a455-063e0d63d7d5-kube-api-access-tw6mb\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk\" (UID: \"f848c811-6e04-4155-a455-063e0d63d7d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:05.048407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:05.048379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f848c811-6e04-4155-a455-063e0d63d7d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk\" (UID: \"f848c811-6e04-4155-a455-063e0d63d7d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:05.048407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:05.048419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw6mb\" (UniqueName: \"kubernetes.io/projected/f848c811-6e04-4155-a455-063e0d63d7d5-kube-api-access-tw6mb\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk\" (UID: \"f848c811-6e04-4155-a455-063e0d63d7d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:05.048746 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:05.048726    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f848c811-6e04-4155-a455-063e0d63d7d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk\" (UID: \"f848c811-6e04-4155-a455-063e0d63d7d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:05.060944 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:05.060924    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw6mb\" (UniqueName: \"kubernetes.io/projected/f848c811-6e04-4155-a455-063e0d63d7d5-kube-api-access-tw6mb\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk\" (UID: \"f848c811-6e04-4155-a455-063e0d63d7d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:05.127897 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:05.127833    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:05.262436 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:05.262372    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"]
Jul 09 18:13:05.266183 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:13:05.266148    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf848c811_6e04_4155_a455_063e0d63d7d5.slice/crio-88675bd20f7fa7d41cec18f960f118970e2154106a3e6c45390766e68e2768b3 WatchSource:0}: Error finding container 88675bd20f7fa7d41cec18f960f118970e2154106a3e6c45390766e68e2768b3: Status 404 returned error can't find the container with id 88675bd20f7fa7d41cec18f960f118970e2154106a3e6c45390766e68e2768b3
Jul 09 18:13:05.997412 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:05.997369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk" event={"ID":"f848c811-6e04-4155-a455-063e0d63d7d5","Type":"ContainerStarted","Data":"88675bd20f7fa7d41cec18f960f118970e2154106a3e6c45390766e68e2768b3"}
Jul 09 18:13:10.014987 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:10.014958    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk" event={"ID":"f848c811-6e04-4155-a455-063e0d63d7d5","Type":"ContainerStarted","Data":"a0784f3cd137a40e12a2f380b4f1712a81916319c9a08a5947d47f1951ef1333"}
Jul 09 18:13:10.015374 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:10.015078    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:10.040662 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:10.040612    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk" podStartSLOduration=2.334866211 podStartE2EDuration="6.040598132s" totalImagesPullingTime="3.705731921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:10.037824219 +0000 UTC m=+585.683817758" watchObservedRunningTime="2026-07-09 18:13:10.040598132 +0000 UTC m=+585.686591676"
Jul 09 18:13:21.020218 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:21.020186    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk"
Jul 09 18:13:24.771830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:24.771801    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:13:24.772728 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:24.772709    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:13:54.033689 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.033612    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-djhsd"]
Jul 09 18:13:54.035793 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.035771    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-djhsd"
Jul 09 18:13:54.039080 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.039061    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zgz47\""
Jul 09 18:13:54.060989 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.060968    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-djhsd"]
Jul 09 18:13:54.126668 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.126641    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfqxg\" (UniqueName: \"kubernetes.io/projected/7d3492a0-64d1-4316-863b-252cd00124f6-kube-api-access-zfqxg\") pod \"authorino-b6c89bb85-djhsd\" (UID: \"7d3492a0-64d1-4316-863b-252cd00124f6\") " pod="kuadrant-system/authorino-b6c89bb85-djhsd"
Jul 09 18:13:54.227651 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.227619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfqxg\" (UniqueName: \"kubernetes.io/projected/7d3492a0-64d1-4316-863b-252cd00124f6-kube-api-access-zfqxg\") pod \"authorino-b6c89bb85-djhsd\" (UID: \"7d3492a0-64d1-4316-863b-252cd00124f6\") " pod="kuadrant-system/authorino-b6c89bb85-djhsd"
Jul 09 18:13:54.254262 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.254234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfqxg\" (UniqueName: \"kubernetes.io/projected/7d3492a0-64d1-4316-863b-252cd00124f6-kube-api-access-zfqxg\") pod \"authorino-b6c89bb85-djhsd\" (UID: \"7d3492a0-64d1-4316-863b-252cd00124f6\") " pod="kuadrant-system/authorino-b6c89bb85-djhsd"
Jul 09 18:13:54.344573 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.344482    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-djhsd"
Jul 09 18:13:54.361267 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.361237    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-ml697"]
Jul 09 18:13:54.363434 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.363409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-ml697"
Jul 09 18:13:54.381614 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.380123    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-ml697"]
Jul 09 18:13:54.429263 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.429187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xblw\" (UniqueName: \"kubernetes.io/projected/94533643-7f72-41bb-9c0e-8108a3e1adb6-kube-api-access-5xblw\") pod \"authorino-8df697c9f-ml697\" (UID: \"94533643-7f72-41bb-9c0e-8108a3e1adb6\") " pod="kuadrant-system/authorino-8df697c9f-ml697"
Jul 09 18:13:54.500578 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.500547    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-djhsd"]
Jul 09 18:13:54.503874 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:13:54.503839    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d3492a0_64d1_4316_863b_252cd00124f6.slice/crio-15282789f653cbfe7b3f0de38ab2b78b4d36b35fb7e86063feeb5d449cce9e92 WatchSource:0}: Error finding container 15282789f653cbfe7b3f0de38ab2b78b4d36b35fb7e86063feeb5d449cce9e92: Status 404 returned error can't find the container with id 15282789f653cbfe7b3f0de38ab2b78b4d36b35fb7e86063feeb5d449cce9e92
Jul 09 18:13:54.530198 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.530168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xblw\" (UniqueName: \"kubernetes.io/projected/94533643-7f72-41bb-9c0e-8108a3e1adb6-kube-api-access-5xblw\") pod \"authorino-8df697c9f-ml697\" (UID: \"94533643-7f72-41bb-9c0e-8108a3e1adb6\") " pod="kuadrant-system/authorino-8df697c9f-ml697"
Jul 09 18:13:54.542977 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.542953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xblw\" (UniqueName: \"kubernetes.io/projected/94533643-7f72-41bb-9c0e-8108a3e1adb6-kube-api-access-5xblw\") pod \"authorino-8df697c9f-ml697\" (UID: \"94533643-7f72-41bb-9c0e-8108a3e1adb6\") " pod="kuadrant-system/authorino-8df697c9f-ml697"
Jul 09 18:13:54.672366 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.672261    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-ml697"
Jul 09 18:13:54.804746 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:54.804719    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-ml697"]
Jul 09 18:13:54.807518 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:13:54.807491    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94533643_7f72_41bb_9c0e_8108a3e1adb6.slice/crio-d4ff0f8a6a3316849f30cfe8113035761e4d12c4e97011a53620598ca9053e08 WatchSource:0}: Error finding container d4ff0f8a6a3316849f30cfe8113035761e4d12c4e97011a53620598ca9053e08: Status 404 returned error can't find the container with id d4ff0f8a6a3316849f30cfe8113035761e4d12c4e97011a53620598ca9053e08
Jul 09 18:13:55.181373 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:55.181313    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-djhsd" event={"ID":"7d3492a0-64d1-4316-863b-252cd00124f6","Type":"ContainerStarted","Data":"15282789f653cbfe7b3f0de38ab2b78b4d36b35fb7e86063feeb5d449cce9e92"}
Jul 09 18:13:55.182388 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:55.182360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-ml697" event={"ID":"94533643-7f72-41bb-9c0e-8108a3e1adb6","Type":"ContainerStarted","Data":"d4ff0f8a6a3316849f30cfe8113035761e4d12c4e97011a53620598ca9053e08"}
Jul 09 18:13:58.197258 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:58.197225    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-djhsd" event={"ID":"7d3492a0-64d1-4316-863b-252cd00124f6","Type":"ContainerStarted","Data":"3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13"}
Jul 09 18:13:58.218075 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:58.217992    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-djhsd" podStartSLOduration=1.154457594 podStartE2EDuration="4.217972293s" totalImagesPullingTime="3.063514699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:58.217168396 +0000 UTC m=+633.863161936" watchObservedRunningTime="2026-07-09 18:13:58.217972293 +0000 UTC m=+633.863965832"
Jul 09 18:13:59.202998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:59.202964    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-ml697" event={"ID":"94533643-7f72-41bb-9c0e-8108a3e1adb6","Type":"ContainerStarted","Data":"bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370"}
Jul 09 18:13:59.228313 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:59.228263    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-ml697" podStartSLOduration=1.626437228 podStartE2EDuration="5.228248287s" totalImagesPullingTime="3.601811059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:59.22679983 +0000 UTC m=+634.872793368" watchObservedRunningTime="2026-07-09 18:13:59.228248287 +0000 UTC m=+634.874241817"
Jul 09 18:13:59.268641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:13:59.268607    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-djhsd"]
Jul 09 18:14:00.207460 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:00.207422    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-djhsd" podUID="7d3492a0-64d1-4316-863b-252cd00124f6" containerName="authorino" containerID="cri-o://3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13" gracePeriod=30
Jul 09 18:14:00.445438 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:00.445416    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-djhsd"
Jul 09 18:14:00.582154 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:00.582062    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfqxg\" (UniqueName: \"kubernetes.io/projected/7d3492a0-64d1-4316-863b-252cd00124f6-kube-api-access-zfqxg\") pod \"7d3492a0-64d1-4316-863b-252cd00124f6\" (UID: \"7d3492a0-64d1-4316-863b-252cd00124f6\") "
Jul 09 18:14:00.584368 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:00.584345    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3492a0-64d1-4316-863b-252cd00124f6-kube-api-access-zfqxg" (OuterVolumeSpecName: "kube-api-access-zfqxg") pod "7d3492a0-64d1-4316-863b-252cd00124f6" (UID: "7d3492a0-64d1-4316-863b-252cd00124f6"). InnerVolumeSpecName "kube-api-access-zfqxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:14:00.683070 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:00.683036    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfqxg\" (UniqueName: \"kubernetes.io/projected/7d3492a0-64d1-4316-863b-252cd00124f6-kube-api-access-zfqxg\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:14:01.212001 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:01.211964    2558 generic.go:356] "Generic (PLEG): container finished" podID="7d3492a0-64d1-4316-863b-252cd00124f6" containerID="3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13" exitCode=0
Jul 09 18:14:01.212001 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:01.212003    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-djhsd"
Jul 09 18:14:01.212469 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:01.212057    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-djhsd" event={"ID":"7d3492a0-64d1-4316-863b-252cd00124f6","Type":"ContainerDied","Data":"3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13"}
Jul 09 18:14:01.212469 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:01.212094    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-djhsd" event={"ID":"7d3492a0-64d1-4316-863b-252cd00124f6","Type":"ContainerDied","Data":"15282789f653cbfe7b3f0de38ab2b78b4d36b35fb7e86063feeb5d449cce9e92"}
Jul 09 18:14:01.212469 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:01.212112    2558 scope.go:117] "RemoveContainer" containerID="3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13"
Jul 09 18:14:01.219941 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:01.219924    2558 scope.go:117] "RemoveContainer" containerID="3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13"
Jul 09 18:14:01.220248 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:14:01.220229    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13\": container with ID starting with 3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13 not found: ID does not exist" containerID="3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13"
Jul 09 18:14:01.220300 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:01.220259    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13"} err="failed to get container status \"3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13\": rpc error: code = NotFound desc = could not find container \"3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13\": container with ID starting with 3492e2f3225728c3a6a305f561671621685288c1036ee71201374a5cc0939a13 not found: ID does not exist"
Jul 09 18:14:01.230756 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:01.230732    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-djhsd"]
Jul 09 18:14:01.234462 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:01.234441    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-djhsd"]
Jul 09 18:14:02.894223 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:02.894187    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d3492a0-64d1-4316-863b-252cd00124f6" path="/var/lib/kubelet/pods/7d3492a0-64d1-4316-863b-252cd00124f6/volumes"
Jul 09 18:14:20.982132 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:20.982099    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-8rb4q"]
Jul 09 18:14:20.984478 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:20.982398    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d3492a0-64d1-4316-863b-252cd00124f6" containerName="authorino"
Jul 09 18:14:20.984478 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:20.982411    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d3492a0-64d1-4316-863b-252cd00124f6" containerName="authorino"
Jul 09 18:14:20.984478 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:20.982461    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d3492a0-64d1-4316-863b-252cd00124f6" containerName="authorino"
Jul 09 18:14:20.985336 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:20.985312    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-8rb4q"
Jul 09 18:14:20.989114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:20.989078    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 09 18:14:20.999807 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:20.999783    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-8rb4q"]
Jul 09 18:14:21.033032 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:21.033004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shdv2\" (UniqueName: \"kubernetes.io/projected/ae920be6-20a5-4809-971f-de2a1417a44b-kube-api-access-shdv2\") pod \"authorino-7c77957575-8rb4q\" (UID: \"ae920be6-20a5-4809-971f-de2a1417a44b\") " pod="kuadrant-system/authorino-7c77957575-8rb4q"
Jul 09 18:14:21.033155 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:21.033049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae920be6-20a5-4809-971f-de2a1417a44b-tls-cert\") pod \"authorino-7c77957575-8rb4q\" (UID: \"ae920be6-20a5-4809-971f-de2a1417a44b\") " pod="kuadrant-system/authorino-7c77957575-8rb4q"
Jul 09 18:14:21.133591 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:21.133559    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shdv2\" (UniqueName: \"kubernetes.io/projected/ae920be6-20a5-4809-971f-de2a1417a44b-kube-api-access-shdv2\") pod \"authorino-7c77957575-8rb4q\" (UID: \"ae920be6-20a5-4809-971f-de2a1417a44b\") " pod="kuadrant-system/authorino-7c77957575-8rb4q"
Jul 09 18:14:21.133723 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:21.133599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae920be6-20a5-4809-971f-de2a1417a44b-tls-cert\") pod \"authorino-7c77957575-8rb4q\" (UID: \"ae920be6-20a5-4809-971f-de2a1417a44b\") " pod="kuadrant-system/authorino-7c77957575-8rb4q"
Jul 09 18:14:21.136242 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:21.136217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae920be6-20a5-4809-971f-de2a1417a44b-tls-cert\") pod \"authorino-7c77957575-8rb4q\" (UID: \"ae920be6-20a5-4809-971f-de2a1417a44b\") " pod="kuadrant-system/authorino-7c77957575-8rb4q"
Jul 09 18:14:21.144750 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:21.144725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shdv2\" (UniqueName: \"kubernetes.io/projected/ae920be6-20a5-4809-971f-de2a1417a44b-kube-api-access-shdv2\") pod \"authorino-7c77957575-8rb4q\" (UID: \"ae920be6-20a5-4809-971f-de2a1417a44b\") " pod="kuadrant-system/authorino-7c77957575-8rb4q"
Jul 09 18:14:21.296440 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:21.296361    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-8rb4q"
Jul 09 18:14:21.424798 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:21.424651    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-8rb4q"]
Jul 09 18:14:21.427266 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:14:21.427234    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae920be6_20a5_4809_971f_de2a1417a44b.slice/crio-e248d30d3c935fd487ef20d7b1de99577b1b83953bcf79c622491eb30e641559 WatchSource:0}: Error finding container e248d30d3c935fd487ef20d7b1de99577b1b83953bcf79c622491eb30e641559: Status 404 returned error can't find the container with id e248d30d3c935fd487ef20d7b1de99577b1b83953bcf79c622491eb30e641559
Jul 09 18:14:22.293007 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:22.292978    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-8rb4q" event={"ID":"ae920be6-20a5-4809-971f-de2a1417a44b","Type":"ContainerStarted","Data":"e248d30d3c935fd487ef20d7b1de99577b1b83953bcf79c622491eb30e641559"}
Jul 09 18:14:23.298442 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:23.298401    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-8rb4q" event={"ID":"ae920be6-20a5-4809-971f-de2a1417a44b","Type":"ContainerStarted","Data":"9ca61efc51f538bff9765de49dd1a5c6a7f06918b94fb79ccff69d76e7c5fd8f"}
Jul 09 18:14:23.332552 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:23.332506    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-8rb4q" podStartSLOduration=2.515143913 podStartE2EDuration="3.332491839s" totalImagesPullingTime="817.347926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:23.331746895 +0000 UTC m=+658.977740434" watchObservedRunningTime="2026-07-09 18:14:23.332491839 +0000 UTC m=+658.978485377"
Jul 09 18:14:23.385937 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:23.385907    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-ml697"]
Jul 09 18:14:23.386110 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:23.386073    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-ml697" podUID="94533643-7f72-41bb-9c0e-8108a3e1adb6" containerName="authorino" containerID="cri-o://bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370" gracePeriod=30
Jul 09 18:14:23.630908 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:23.630884    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-ml697"
Jul 09 18:14:23.650673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:23.650640    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xblw\" (UniqueName: \"kubernetes.io/projected/94533643-7f72-41bb-9c0e-8108a3e1adb6-kube-api-access-5xblw\") pod \"94533643-7f72-41bb-9c0e-8108a3e1adb6\" (UID: \"94533643-7f72-41bb-9c0e-8108a3e1adb6\") "
Jul 09 18:14:23.653001 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:23.652973    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94533643-7f72-41bb-9c0e-8108a3e1adb6-kube-api-access-5xblw" (OuterVolumeSpecName: "kube-api-access-5xblw") pod "94533643-7f72-41bb-9c0e-8108a3e1adb6" (UID: "94533643-7f72-41bb-9c0e-8108a3e1adb6"). InnerVolumeSpecName "kube-api-access-5xblw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:14:23.751770 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:23.751733    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xblw\" (UniqueName: \"kubernetes.io/projected/94533643-7f72-41bb-9c0e-8108a3e1adb6-kube-api-access-5xblw\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:14:24.303539 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.303506    2558 generic.go:356] "Generic (PLEG): container finished" podID="94533643-7f72-41bb-9c0e-8108a3e1adb6" containerID="bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370" exitCode=0
Jul 09 18:14:24.304016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.303547    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-ml697"
Jul 09 18:14:24.304016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.303591    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-ml697" event={"ID":"94533643-7f72-41bb-9c0e-8108a3e1adb6","Type":"ContainerDied","Data":"bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370"}
Jul 09 18:14:24.304016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.303627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-ml697" event={"ID":"94533643-7f72-41bb-9c0e-8108a3e1adb6","Type":"ContainerDied","Data":"d4ff0f8a6a3316849f30cfe8113035761e4d12c4e97011a53620598ca9053e08"}
Jul 09 18:14:24.304016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.303642    2558 scope.go:117] "RemoveContainer" containerID="bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370"
Jul 09 18:14:24.312124 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.312109    2558 scope.go:117] "RemoveContainer" containerID="bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370"
Jul 09 18:14:24.312376 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:14:24.312357    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370\": container with ID starting with bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370 not found: ID does not exist" containerID="bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370"
Jul 09 18:14:24.312420 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.312387    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370"} err="failed to get container status \"bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370\": rpc error: code = NotFound desc = could not find container \"bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370\": container with ID starting with bb54598e36adf42af6344f6d1c841a7c0126028021a181b9941aad40ff661370 not found: ID does not exist"
Jul 09 18:14:24.327954 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.327928    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-ml697"]
Jul 09 18:14:24.330650 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.330631    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-ml697"]
Jul 09 18:14:24.892880 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:14:24.892845    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94533643-7f72-41bb-9c0e-8108a3e1adb6" path="/var/lib/kubelet/pods/94533643-7f72-41bb-9c0e-8108a3e1adb6/volumes"
Jul 09 18:15:16.732708 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.732615    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-hp242"]
Jul 09 18:15:16.733261 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.733009    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94533643-7f72-41bb-9c0e-8108a3e1adb6" containerName="authorino"
Jul 09 18:15:16.733261 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.733026    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="94533643-7f72-41bb-9c0e-8108a3e1adb6" containerName="authorino"
Jul 09 18:15:16.733261 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.733117    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="94533643-7f72-41bb-9c0e-8108a3e1adb6" containerName="authorino"
Jul 09 18:15:16.736163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.736141    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hp242"
Jul 09 18:15:16.739504 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.739486    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:15:16.739611 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.739486    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qftwd\""
Jul 09 18:15:16.741097 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.741083    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 18:15:16.741182 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.741166    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:15:16.745452 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.745431    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hp242"]
Jul 09 18:15:16.849785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.849753    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805d820-0233-4118-bc76-20e1c70e108c-model-cache\") pod \"s3-init-hp242\" (UID: \"0805d820-0233-4118-bc76-20e1c70e108c\") " pod="kserve/s3-init-hp242"
Jul 09 18:15:16.849785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.849789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k98bc\" (UniqueName: \"kubernetes.io/projected/0805d820-0233-4118-bc76-20e1c70e108c-kube-api-access-k98bc\") pod \"s3-init-hp242\" (UID: \"0805d820-0233-4118-bc76-20e1c70e108c\") " pod="kserve/s3-init-hp242"
Jul 09 18:15:16.950726 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.950692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805d820-0233-4118-bc76-20e1c70e108c-model-cache\") pod \"s3-init-hp242\" (UID: \"0805d820-0233-4118-bc76-20e1c70e108c\") " pod="kserve/s3-init-hp242"
Jul 09 18:15:16.950968 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.950732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k98bc\" (UniqueName: \"kubernetes.io/projected/0805d820-0233-4118-bc76-20e1c70e108c-kube-api-access-k98bc\") pod \"s3-init-hp242\" (UID: \"0805d820-0233-4118-bc76-20e1c70e108c\") " pod="kserve/s3-init-hp242"
Jul 09 18:15:16.951054 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.951033    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805d820-0233-4118-bc76-20e1c70e108c-model-cache\") pod \"s3-init-hp242\" (UID: \"0805d820-0233-4118-bc76-20e1c70e108c\") " pod="kserve/s3-init-hp242"
Jul 09 18:15:16.960311 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:16.960281    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k98bc\" (UniqueName: \"kubernetes.io/projected/0805d820-0233-4118-bc76-20e1c70e108c-kube-api-access-k98bc\") pod \"s3-init-hp242\" (UID: \"0805d820-0233-4118-bc76-20e1c70e108c\") " pod="kserve/s3-init-hp242"
Jul 09 18:15:17.045388 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:17.045279    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hp242"
Jul 09 18:15:17.192011 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:17.191876    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hp242"]
Jul 09 18:15:17.194611 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:15:17.194581    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0805d820_0233_4118_bc76_20e1c70e108c.slice/crio-5ae8b55f1c5240327c0d9bae7a60146f3cb4cb5cef3381190e3f45bfe7cde192 WatchSource:0}: Error finding container 5ae8b55f1c5240327c0d9bae7a60146f3cb4cb5cef3381190e3f45bfe7cde192: Status 404 returned error can't find the container with id 5ae8b55f1c5240327c0d9bae7a60146f3cb4cb5cef3381190e3f45bfe7cde192
Jul 09 18:15:17.502063 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:17.502033    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hp242" event={"ID":"0805d820-0233-4118-bc76-20e1c70e108c","Type":"ContainerStarted","Data":"5ae8b55f1c5240327c0d9bae7a60146f3cb4cb5cef3381190e3f45bfe7cde192"}
Jul 09 18:15:22.524610 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:22.524574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hp242" event={"ID":"0805d820-0233-4118-bc76-20e1c70e108c","Type":"ContainerStarted","Data":"30b3d53d6139d261c76fafe8c2664f04b7e8ebabe9b652129253b7db705185fa"}
Jul 09 18:15:24.533906 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:24.533867    2558 generic.go:356] "Generic (PLEG): container finished" podID="0805d820-0233-4118-bc76-20e1c70e108c" containerID="30b3d53d6139d261c76fafe8c2664f04b7e8ebabe9b652129253b7db705185fa" exitCode=0
Jul 09 18:15:24.534372 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:24.533939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hp242" event={"ID":"0805d820-0233-4118-bc76-20e1c70e108c","Type":"ContainerDied","Data":"30b3d53d6139d261c76fafe8c2664f04b7e8ebabe9b652129253b7db705185fa"}
Jul 09 18:15:29.269622 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:29.269602    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 18:15:29.558540 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:29.558451    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hp242" event={"ID":"0805d820-0233-4118-bc76-20e1c70e108c","Type":"ContainerStarted","Data":"831fa1814d6ff68ba60c9c9303f82be088777da366745047871bb4e6b084f07c"}
Jul 09 18:15:29.577611 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:15:29.577566    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-hp242" podStartSLOduration=2.48923921 podStartE2EDuration="13.577550333s" totalImagesPullingTime="9.332058189s" totalInitContainerRuntime="1.756252934s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:29.576684266 +0000 UTC m=+725.222677807" watchObservedRunningTime="2026-07-09 18:15:29.577550333 +0000 UTC m=+725.223543876"
Jul 09 18:16:34.811321 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:34.811237    2558 generic.go:356] "Generic (PLEG): container finished" podID="0805d820-0233-4118-bc76-20e1c70e108c" containerID="831fa1814d6ff68ba60c9c9303f82be088777da366745047871bb4e6b084f07c" exitCode=0
Jul 09 18:16:34.811321 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:34.811285    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hp242" event={"ID":"0805d820-0233-4118-bc76-20e1c70e108c","Type":"ContainerDied","Data":"831fa1814d6ff68ba60c9c9303f82be088777da366745047871bb4e6b084f07c"}
Jul 09 18:16:35.942945 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:35.942920    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hp242"
Jul 09 18:16:36.041095 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:36.041062    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805d820-0233-4118-bc76-20e1c70e108c-model-cache\") pod \"0805d820-0233-4118-bc76-20e1c70e108c\" (UID: \"0805d820-0233-4118-bc76-20e1c70e108c\") "
Jul 09 18:16:36.041278 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:36.041112    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k98bc\" (UniqueName: \"kubernetes.io/projected/0805d820-0233-4118-bc76-20e1c70e108c-kube-api-access-k98bc\") pod \"0805d820-0233-4118-bc76-20e1c70e108c\" (UID: \"0805d820-0233-4118-bc76-20e1c70e108c\") "
Jul 09 18:16:36.043322 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:36.043300    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0805d820-0233-4118-bc76-20e1c70e108c-kube-api-access-k98bc" (OuterVolumeSpecName: "kube-api-access-k98bc") pod "0805d820-0233-4118-bc76-20e1c70e108c" (UID: "0805d820-0233-4118-bc76-20e1c70e108c"). InnerVolumeSpecName "kube-api-access-k98bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:16:36.063211 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:36.063188    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0805d820-0233-4118-bc76-20e1c70e108c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0805d820-0233-4118-bc76-20e1c70e108c" (UID: "0805d820-0233-4118-bc76-20e1c70e108c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:36.141941 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:36.141867    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0805d820-0233-4118-bc76-20e1c70e108c-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:16:36.141941 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:36.141907    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k98bc\" (UniqueName: \"kubernetes.io/projected/0805d820-0233-4118-bc76-20e1c70e108c-kube-api-access-k98bc\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:16:36.821010 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:36.820974    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hp242" event={"ID":"0805d820-0233-4118-bc76-20e1c70e108c","Type":"ContainerDied","Data":"5ae8b55f1c5240327c0d9bae7a60146f3cb4cb5cef3381190e3f45bfe7cde192"}
Jul 09 18:16:36.821010 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:36.821001    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hp242"
Jul 09 18:16:36.821010 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:16:36.821015    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ae8b55f1c5240327c0d9bae7a60146f3cb4cb5cef3381190e3f45bfe7cde192"
Jul 09 18:17:00.766558 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.766521    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"]
Jul 09 18:17:00.766967 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.766892    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0805d820-0233-4118-bc76-20e1c70e108c" containerName="download-hf-model"
Jul 09 18:17:00.766967 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.766910    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0805d820-0233-4118-bc76-20e1c70e108c" containerName="download-hf-model"
Jul 09 18:17:00.766967 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.766919    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0805d820-0233-4118-bc76-20e1c70e108c" containerName="s3-init"
Jul 09 18:17:00.766967 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.766927    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0805d820-0233-4118-bc76-20e1c70e108c" containerName="s3-init"
Jul 09 18:17:00.767163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.767003    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0805d820-0233-4118-bc76-20e1c70e108c" containerName="s3-init"
Jul 09 18:17:00.769768 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.769748    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:00.773060 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.773040    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:17:00.773189 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.773040    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:17:00.774805 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.774785    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:17:00.774805 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.774798    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 09 18:17:00.780102 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.780078    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"]
Jul 09 18:17:00.939024 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.938991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:00.939200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.939036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:00.939200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.939055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:00.939200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.939154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9e7f62-8347-4d04-a228-69f26591ca54-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:00.939200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.939191    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpl8p\" (UniqueName: \"kubernetes.io/projected/9c9e7f62-8347-4d04-a228-69f26591ca54-kube-api-access-zpl8p\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:00.939549 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:00.939214    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.039641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.039556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpl8p\" (UniqueName: \"kubernetes.io/projected/9c9e7f62-8347-4d04-a228-69f26591ca54-kube-api-access-zpl8p\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.039641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.039591    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.039641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.039622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.039903 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.039668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.039903 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.039764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.039903 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.039837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9e7f62-8347-4d04-a228-69f26591ca54-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.040154 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.040116    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.040154 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.040151    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.040533 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.040203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.042098 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.042070    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.042494 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.042471    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9e7f62-8347-4d04-a228-69f26591ca54-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.054295 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.054264    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpl8p\" (UniqueName: \"kubernetes.io/projected/9c9e7f62-8347-4d04-a228-69f26591ca54-kube-api-access-zpl8p\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.081003 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.080972    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:01.224559 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.224525    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"]
Jul 09 18:17:01.227380 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:17:01.227356    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9e7f62_8347_4d04_a228_69f26591ca54.slice/crio-a0e499b0d27887770047f6ff8c1603b3d23e7903bf79a3821eb79c3aacd62335 WatchSource:0}: Error finding container a0e499b0d27887770047f6ff8c1603b3d23e7903bf79a3821eb79c3aacd62335: Status 404 returned error can't find the container with id a0e499b0d27887770047f6ff8c1603b3d23e7903bf79a3821eb79c3aacd62335
Jul 09 18:17:01.229151 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.229134    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:17:01.913802 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:01.913771    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq" event={"ID":"9c9e7f62-8347-4d04-a228-69f26591ca54","Type":"ContainerStarted","Data":"a0e499b0d27887770047f6ff8c1603b3d23e7903bf79a3821eb79c3aacd62335"}
Jul 09 18:17:02.918451 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:02.918408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq" event={"ID":"9c9e7f62-8347-4d04-a228-69f26591ca54","Type":"ContainerStarted","Data":"35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b"}
Jul 09 18:17:02.938113 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:02.938061    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq" podStartSLOduration=1.8576834679999998 podStartE2EDuration="2.938045197s" totalImagesPullingTime="1.080361729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:02.937299039 +0000 UTC m=+818.583292581" watchObservedRunningTime="2026-07-09 18:17:02.938045197 +0000 UTC m=+818.584038751"
Jul 09 18:17:03.081103 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:03.081061    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:11.082084 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:11.082050    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:11.094525 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:11.094502    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:11.963716 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:11.963683    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:29.211861 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.211830    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"]
Jul 09 18:17:29.215017 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.214993    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.218442 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.218423    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:17:29.218560 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.218497    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 09 18:17:29.227320 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.227294    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"]
Jul 09 18:17:29.253703 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.253657    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-dshm\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.253870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.253728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-home\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.253870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.253752    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c2bed-254e-4bc8-9028-46e495bb3070-tls-certs\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.253870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.253773    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-model-cache\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.253870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.253841    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktcc\" (UniqueName: \"kubernetes.io/projected/c62c2bed-254e-4bc8-9028-46e495bb3070-kube-api-access-gktcc\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.254005 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.253875    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.254005 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.253894    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.354767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.354728    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktcc\" (UniqueName: \"kubernetes.io/projected/c62c2bed-254e-4bc8-9028-46e495bb3070-kube-api-access-gktcc\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.354957 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.354779    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.354957 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.354805    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.354957 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.354831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-dshm\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.354957 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.354874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-home\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.354957 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.354897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c2bed-254e-4bc8-9028-46e495bb3070-tls-certs\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.355198 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.355033    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-model-cache\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.355422 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.355389    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.355546 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.355427    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.355546 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.355460    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-home\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.355546 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.355497    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-model-cache\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.357463 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.357439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-dshm\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.357825 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.357802    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c2bed-254e-4bc8-9028-46e495bb3070-tls-certs\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.371668 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.371642    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktcc\" (UniqueName: \"kubernetes.io/projected/c62c2bed-254e-4bc8-9028-46e495bb3070-kube-api-access-gktcc\") pod \"precise-prefix-cache-test-kserve-5bc9974d7f-qng2q\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.524804 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.524707    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:29.658230 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:29.658205    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"]
Jul 09 18:17:29.660502 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:17:29.660459    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62c2bed_254e_4bc8_9028_46e495bb3070.slice/crio-22b206c0ffd2a8649dc464fe70bc9f77dc759d90ee411071f924c61c4ffa243a WatchSource:0}: Error finding container 22b206c0ffd2a8649dc464fe70bc9f77dc759d90ee411071f924c61c4ffa243a: Status 404 returned error can't find the container with id 22b206c0ffd2a8649dc464fe70bc9f77dc759d90ee411071f924c61c4ffa243a
Jul 09 18:17:30.027235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:30.027192    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q" event={"ID":"c62c2bed-254e-4bc8-9028-46e495bb3070","Type":"ContainerStarted","Data":"9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063"}
Jul 09 18:17:30.027413 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:30.027285    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q" event={"ID":"c62c2bed-254e-4bc8-9028-46e495bb3070","Type":"ContainerStarted","Data":"22b206c0ffd2a8649dc464fe70bc9f77dc759d90ee411071f924c61c4ffa243a"}
Jul 09 18:17:31.875458 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:31.875421    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"]
Jul 09 18:17:34.045404 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:34.045370    2558 generic.go:356] "Generic (PLEG): container finished" podID="c62c2bed-254e-4bc8-9028-46e495bb3070" containerID="9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063" exitCode=0
Jul 09 18:17:34.045867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:34.045454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q" event={"ID":"c62c2bed-254e-4bc8-9028-46e495bb3070","Type":"ContainerDied","Data":"9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063"}
Jul 09 18:17:35.051273 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:35.051236    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q" event={"ID":"c62c2bed-254e-4bc8-9028-46e495bb3070","Type":"ContainerStarted","Data":"b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb"}
Jul 09 18:17:35.090447 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:35.090399    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q" podStartSLOduration=2.26282569 podStartE2EDuration="6.090384462s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.827558772s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:35.089497734 +0000 UTC m=+850.735491272" watchObservedRunningTime="2026-07-09 18:17:35.090384462 +0000 UTC m=+850.736378001"
Jul 09 18:17:35.525706 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:35.525671    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:39.524938 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:39.524902    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:39.537838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:39.537804    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:40.081957 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:40.081929    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:17:44.639463 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.639432    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"]
Jul 09 18:17:44.659002 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.658971    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.659318 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.659290    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"]
Jul 09 18:17:44.662405 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.662384    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-mg5cs\""
Jul 09 18:17:44.662535 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.662385    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:17:44.784412 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.784380    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.784583 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.784424    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.784583 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.784449    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gs9j\" (UniqueName: \"kubernetes.io/projected/35bb2f54-cb55-4392-8f28-8b2709b69211-kube-api-access-8gs9j\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.784583 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.784534    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.784583 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.784571    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.784745 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.784608    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.784745 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.784633    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.784745 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.784647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bb2f54-cb55-4392-8f28-8b2709b69211-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.784862 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.784742    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885420 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885420 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885452    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885473    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gs9j\" (UniqueName: \"kubernetes.io/projected/35bb2f54-cb55-4392-8f28-8b2709b69211-kube-api-access-8gs9j\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885645    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885680    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bb2f54-cb55-4392-8f28-8b2709b69211-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.885867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885836    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.886019 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.885902    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.886057 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.886023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.886090 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.886057    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.886433 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.886411    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bb2f54-cb55-4392-8f28-8b2709b69211-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.888432 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.888409    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.888519 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.888494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.897084 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.897016    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bb2f54-cb55-4392-8f28-8b2709b69211-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.897274 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.897254    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gs9j\" (UniqueName: \"kubernetes.io/projected/35bb2f54-cb55-4392-8f28-8b2709b69211-kube-api-access-8gs9j\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-755f5\" (UID: \"35bb2f54-cb55-4392-8f28-8b2709b69211\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:44.970678 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:44.970653    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:45.117732 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:45.117709    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"]
Jul 09 18:17:45.119673 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:17:45.119648    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bb2f54_cb55_4392_8f28_8b2709b69211.slice/crio-c7fcfe0afed97f8f89391b15a7c135c9b67562ca94324d102d31a834e7db479c WatchSource:0}: Error finding container c7fcfe0afed97f8f89391b15a7c135c9b67562ca94324d102d31a834e7db479c: Status 404 returned error can't find the container with id c7fcfe0afed97f8f89391b15a7c135c9b67562ca94324d102d31a834e7db479c
Jul 09 18:17:46.097109 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:46.097063    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" event={"ID":"35bb2f54-cb55-4392-8f28-8b2709b69211","Type":"ContainerStarted","Data":"c7fcfe0afed97f8f89391b15a7c135c9b67562ca94324d102d31a834e7db479c"}
Jul 09 18:17:46.897896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:46.897848    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq" podUID="9c9e7f62-8347-4d04-a228-69f26591ca54" containerName="main" containerID="cri-o://35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b" gracePeriod=45
Jul 09 18:17:46.953609 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:46.953561    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq" podUID="9c9e7f62-8347-4d04-a228-69f26591ca54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.23:8000/health\": dial tcp 10.133.0.23:8000: connect: connection refused"
Jul 09 18:17:47.800452 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:47.800399    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 18:17:47.800831 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:47.800486    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 18:17:47.800831 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:47.800514    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 18:17:47.916413 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:47.916391    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:48.013526 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.013493    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-model-cache\") pod \"9c9e7f62-8347-4d04-a228-69f26591ca54\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") "
Jul 09 18:17:48.013753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.013548    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpl8p\" (UniqueName: \"kubernetes.io/projected/9c9e7f62-8347-4d04-a228-69f26591ca54-kube-api-access-zpl8p\") pod \"9c9e7f62-8347-4d04-a228-69f26591ca54\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") "
Jul 09 18:17:48.013753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.013588    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-tmp-dir\") pod \"9c9e7f62-8347-4d04-a228-69f26591ca54\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") "
Jul 09 18:17:48.013753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.013610    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-home\") pod \"9c9e7f62-8347-4d04-a228-69f26591ca54\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") "
Jul 09 18:17:48.013753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.013638    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-dshm\") pod \"9c9e7f62-8347-4d04-a228-69f26591ca54\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") "
Jul 09 18:17:48.013753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.013712    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9e7f62-8347-4d04-a228-69f26591ca54-tls-certs\") pod \"9c9e7f62-8347-4d04-a228-69f26591ca54\" (UID: \"9c9e7f62-8347-4d04-a228-69f26591ca54\") "
Jul 09 18:17:48.014015 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.013836    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c9e7f62-8347-4d04-a228-69f26591ca54" (UID: "9c9e7f62-8347-4d04-a228-69f26591ca54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:48.014015 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.013874    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-home" (OuterVolumeSpecName: "home") pod "9c9e7f62-8347-4d04-a228-69f26591ca54" (UID: "9c9e7f62-8347-4d04-a228-69f26591ca54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:48.014133 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.014103    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c9e7f62-8347-4d04-a228-69f26591ca54" (UID: "9c9e7f62-8347-4d04-a228-69f26591ca54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:48.014192 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.014108    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:17:48.014192 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.014159    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:17:48.016253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.016226    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9e7f62-8347-4d04-a228-69f26591ca54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c9e7f62-8347-4d04-a228-69f26591ca54" (UID: "9c9e7f62-8347-4d04-a228-69f26591ca54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:17:48.016253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.016239    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9e7f62-8347-4d04-a228-69f26591ca54-kube-api-access-zpl8p" (OuterVolumeSpecName: "kube-api-access-zpl8p") pod "9c9e7f62-8347-4d04-a228-69f26591ca54" (UID: "9c9e7f62-8347-4d04-a228-69f26591ca54"). InnerVolumeSpecName "kube-api-access-zpl8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:48.016453 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.016294    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-dshm" (OuterVolumeSpecName: "dshm") pod "9c9e7f62-8347-4d04-a228-69f26591ca54" (UID: "9c9e7f62-8347-4d04-a228-69f26591ca54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:48.107196 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.107158    2558 generic.go:356] "Generic (PLEG): container finished" podID="9c9e7f62-8347-4d04-a228-69f26591ca54" containerID="35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b" exitCode=0
Jul 09 18:17:48.107407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.107212    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"
Jul 09 18:17:48.107407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.107253    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq" event={"ID":"9c9e7f62-8347-4d04-a228-69f26591ca54","Type":"ContainerDied","Data":"35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b"}
Jul 09 18:17:48.107407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.107317    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq" event={"ID":"9c9e7f62-8347-4d04-a228-69f26591ca54","Type":"ContainerDied","Data":"a0e499b0d27887770047f6ff8c1603b3d23e7903bf79a3821eb79c3aacd62335"}
Jul 09 18:17:48.107407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.107366    2558 scope.go:117] "RemoveContainer" containerID="35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b"
Jul 09 18:17:48.108923 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.108893    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" event={"ID":"35bb2f54-cb55-4392-8f28-8b2709b69211","Type":"ContainerStarted","Data":"305104d3d71408425e4382d8d6f24bfddbe98330afb2c8c820bbc364ee2f3436"}
Jul 09 18:17:48.114636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.114612    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:17:48.114751 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.114638    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9e7f62-8347-4d04-a228-69f26591ca54-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:17:48.114751 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.114653    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9e7f62-8347-4d04-a228-69f26591ca54-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:17:48.114751 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.114666    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpl8p\" (UniqueName: \"kubernetes.io/projected/9c9e7f62-8347-4d04-a228-69f26591ca54-kube-api-access-zpl8p\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:17:48.116476 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.116457    2558 scope.go:117] "RemoveContainer" containerID="35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b"
Jul 09 18:17:48.116733 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:17:48.116714    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b\": container with ID starting with 35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b not found: ID does not exist" containerID="35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b"
Jul 09 18:17:48.116803 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.116739    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b"} err="failed to get container status \"35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b\": rpc error: code = NotFound desc = could not find container \"35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b\": container with ID starting with 35cd11dc62e8dce79a0d3cfc71d119db4f0c9fb9d4166f2f5137f84901a40b0b not found: ID does not exist"
Jul 09 18:17:48.143664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.143623    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" podStartSLOduration=1.465178801 podStartE2EDuration="4.143608966s" totalImagesPullingTime="2.678430165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:48.142320286 +0000 UTC m=+863.788313826" watchObservedRunningTime="2026-07-09 18:17:48.143608966 +0000 UTC m=+863.789602504"
Jul 09 18:17:48.169041 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.169014    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"]
Jul 09 18:17:48.172971 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.172948    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54flvq"]
Jul 09 18:17:48.893611 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.893577    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9e7f62-8347-4d04-a228-69f26591ca54" path="/var/lib/kubelet/pods/9c9e7f62-8347-4d04-a228-69f26591ca54/volumes"
Jul 09 18:17:48.971710 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.971674    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:48.976313 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:48.976290    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:49.118196 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:49.118161    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:17:49.119021 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:17:49.119004    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:18:14.906782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.906752    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"]
Jul 09 18:18:14.907198 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.907018    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9e7f62-8347-4d04-a228-69f26591ca54" containerName="main"
Jul 09 18:18:14.907198 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.907030    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9e7f62-8347-4d04-a228-69f26591ca54" containerName="main"
Jul 09 18:18:14.907198 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.907087    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9e7f62-8347-4d04-a228-69f26591ca54" containerName="main"
Jul 09 18:18:14.911430 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.911415    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:14.915045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.915022    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 09 18:18:14.929487 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.929462    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"]
Jul 09 18:18:14.943986 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.943960    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qnb\" (UniqueName: \"kubernetes.io/projected/4d9e9497-5a5d-4363-a506-c7cffed08de0-kube-api-access-l8qnb\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:14.944104 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.943997    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-tmp-dir\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:14.944104 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.944030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-kserve-provision-location\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:14.944195 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.944128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-home\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:14.944195 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.944170    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-dshm\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:14.944287 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.944214    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9e9497-5a5d-4363-a506-c7cffed08de0-tls-certs\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:14.944356 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:14.944289    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-model-cache\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.045665 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.045632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-kserve-provision-location\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.045824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.045678    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-home\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.045824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.045695    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-dshm\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.045824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.045719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9e9497-5a5d-4363-a506-c7cffed08de0-tls-certs\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.045824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.045740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-model-cache\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.045824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.045777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qnb\" (UniqueName: \"kubernetes.io/projected/4d9e9497-5a5d-4363-a506-c7cffed08de0-kube-api-access-l8qnb\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.045824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.045814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-tmp-dir\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.046086 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.046061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-kserve-provision-location\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.046369 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.046348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-home\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.046634 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.046435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-model-cache\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.046634 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.046443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-tmp-dir\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.048200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.048178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-dshm\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.048364 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.048347    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9e9497-5a5d-4363-a506-c7cffed08de0-tls-certs\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.057297 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.057269    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qnb\" (UniqueName: \"kubernetes.io/projected/4d9e9497-5a5d-4363-a506-c7cffed08de0-kube-api-access-l8qnb\") pod \"auth-enabled-test-kserve-865fb4fc77-6mq9w\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.252246 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.252208    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:15.398091 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:15.398066    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"]
Jul 09 18:18:15.399813 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:18:15.399784    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9e9497_5a5d_4363_a506_c7cffed08de0.slice/crio-161f0363311d4b6a82f5975194a5a6681164939ae15dbf65655161f00494b14b WatchSource:0}: Error finding container 161f0363311d4b6a82f5975194a5a6681164939ae15dbf65655161f00494b14b: Status 404 returned error can't find the container with id 161f0363311d4b6a82f5975194a5a6681164939ae15dbf65655161f00494b14b
Jul 09 18:18:16.227961 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:16.227926    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" event={"ID":"4d9e9497-5a5d-4363-a506-c7cffed08de0","Type":"ContainerStarted","Data":"c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16"}
Jul 09 18:18:16.228364 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:16.227986    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" event={"ID":"4d9e9497-5a5d-4363-a506-c7cffed08de0","Type":"ContainerStarted","Data":"161f0363311d4b6a82f5975194a5a6681164939ae15dbf65655161f00494b14b"}
Jul 09 18:18:19.249290 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:19.249254    2558 generic.go:356] "Generic (PLEG): container finished" podID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerID="c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16" exitCode=0
Jul 09 18:18:19.249696 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:19.249339    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" event={"ID":"4d9e9497-5a5d-4363-a506-c7cffed08de0","Type":"ContainerDied","Data":"c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16"}
Jul 09 18:18:24.802640 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:24.802608    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:18:24.803573 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:24.803548    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:18:42.275809 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:42.275774    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"]
Jul 09 18:18:50.404952 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:50.404915    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" event={"ID":"4d9e9497-5a5d-4363-a506-c7cffed08de0","Type":"ContainerStarted","Data":"8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339"}
Jul 09 18:18:50.432652 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:50.432589    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podStartSLOduration=2.22195746 podStartE2EDuration="36.432568408s" totalImagesPullingTime="30.985860171s" totalInitContainerRuntime="3.224750777s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:50.429779397 +0000 UTC m=+926.075772939" watchObservedRunningTime="2026-07-09 18:18:50.432568408 +0000 UTC m=+926.078561948"
Jul 09 18:18:51.252774 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:51.252734    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:55.252621 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:55.252577    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:18:55.253921 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:55.253895    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:18:57.390186 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:57.390143    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q" podUID="c62c2bed-254e-4bc8-9028-46e495bb3070" containerName="main" containerID="cri-o://b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb" gracePeriod=45
Jul 09 18:18:58.071694 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.071653    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q" podUID="c62c2bed-254e-4bc8-9028-46e495bb3070" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": dial tcp 10.133.0.24:8000: connect: connection refused"
Jul 09 18:18:58.198523 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.198490    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:18:58.233307 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233275    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-kserve-provision-location\") pod \"c62c2bed-254e-4bc8-9028-46e495bb3070\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") "
Jul 09 18:18:58.233506 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233359    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-dshm\") pod \"c62c2bed-254e-4bc8-9028-46e495bb3070\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") "
Jul 09 18:18:58.233506 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233389    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c2bed-254e-4bc8-9028-46e495bb3070-tls-certs\") pod \"c62c2bed-254e-4bc8-9028-46e495bb3070\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") "
Jul 09 18:18:58.233506 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233438    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-tmp-dir\") pod \"c62c2bed-254e-4bc8-9028-46e495bb3070\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") "
Jul 09 18:18:58.233666 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233564    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-home\") pod \"c62c2bed-254e-4bc8-9028-46e495bb3070\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") "
Jul 09 18:18:58.233666 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233639    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gktcc\" (UniqueName: \"kubernetes.io/projected/c62c2bed-254e-4bc8-9028-46e495bb3070-kube-api-access-gktcc\") pod \"c62c2bed-254e-4bc8-9028-46e495bb3070\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") "
Jul 09 18:18:58.233776 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233661    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c62c2bed-254e-4bc8-9028-46e495bb3070" (UID: "c62c2bed-254e-4bc8-9028-46e495bb3070"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:58.233776 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233686    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-model-cache\") pod \"c62c2bed-254e-4bc8-9028-46e495bb3070\" (UID: \"c62c2bed-254e-4bc8-9028-46e495bb3070\") "
Jul 09 18:18:58.233874 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233814    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-home" (OuterVolumeSpecName: "home") pod "c62c2bed-254e-4bc8-9028-46e495bb3070" (UID: "c62c2bed-254e-4bc8-9028-46e495bb3070"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:58.234007 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233957    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:18:58.234007 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.233980    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:18:58.234304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.234085    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-model-cache" (OuterVolumeSpecName: "model-cache") pod "c62c2bed-254e-4bc8-9028-46e495bb3070" (UID: "c62c2bed-254e-4bc8-9028-46e495bb3070"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:58.235727 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.235699    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-dshm" (OuterVolumeSpecName: "dshm") pod "c62c2bed-254e-4bc8-9028-46e495bb3070" (UID: "c62c2bed-254e-4bc8-9028-46e495bb3070"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:58.236256 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.236230    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62c2bed-254e-4bc8-9028-46e495bb3070-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c62c2bed-254e-4bc8-9028-46e495bb3070" (UID: "c62c2bed-254e-4bc8-9028-46e495bb3070"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:58.236408 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.236319    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62c2bed-254e-4bc8-9028-46e495bb3070-kube-api-access-gktcc" (OuterVolumeSpecName: "kube-api-access-gktcc") pod "c62c2bed-254e-4bc8-9028-46e495bb3070" (UID: "c62c2bed-254e-4bc8-9028-46e495bb3070"). InnerVolumeSpecName "kube-api-access-gktcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:58.289616 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.289513    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c62c2bed-254e-4bc8-9028-46e495bb3070" (UID: "c62c2bed-254e-4bc8-9028-46e495bb3070"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:58.334866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.334828    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gktcc\" (UniqueName: \"kubernetes.io/projected/c62c2bed-254e-4bc8-9028-46e495bb3070-kube-api-access-gktcc\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:18:58.334866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.334854    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:18:58.334866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.334863    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:18:58.334866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.334875    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c2bed-254e-4bc8-9028-46e495bb3070-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:18:58.334866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.334883    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c2bed-254e-4bc8-9028-46e495bb3070-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:18:58.442042 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.442004    2558 generic.go:356] "Generic (PLEG): container finished" podID="c62c2bed-254e-4bc8-9028-46e495bb3070" containerID="b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb" exitCode=0
Jul 09 18:18:58.442555 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.442059    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"
Jul 09 18:18:58.442555 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.442097    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q" event={"ID":"c62c2bed-254e-4bc8-9028-46e495bb3070","Type":"ContainerDied","Data":"b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb"}
Jul 09 18:18:58.442555 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.442155    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q" event={"ID":"c62c2bed-254e-4bc8-9028-46e495bb3070","Type":"ContainerDied","Data":"22b206c0ffd2a8649dc464fe70bc9f77dc759d90ee411071f924c61c4ffa243a"}
Jul 09 18:18:58.442555 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.442179    2558 scope.go:117] "RemoveContainer" containerID="b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb"
Jul 09 18:18:58.451157 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.451143    2558 scope.go:117] "RemoveContainer" containerID="9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063"
Jul 09 18:18:58.458642 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.458624    2558 scope.go:117] "RemoveContainer" containerID="b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb"
Jul 09 18:18:58.458913 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:18:58.458894    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb\": container with ID starting with b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb not found: ID does not exist" containerID="b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb"
Jul 09 18:18:58.458966 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.458921    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb"} err="failed to get container status \"b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb\": rpc error: code = NotFound desc = could not find container \"b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb\": container with ID starting with b2e688f0c9f1d487af1c7c7ec7e501c3858a7c395d1d2a456a1c2a4c619a60bb not found: ID does not exist"
Jul 09 18:18:58.458966 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.458938    2558 scope.go:117] "RemoveContainer" containerID="9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063"
Jul 09 18:18:58.459212 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:18:58.459190    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063\": container with ID starting with 9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063 not found: ID does not exist" containerID="9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063"
Jul 09 18:18:58.459281 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.459225    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063"} err="failed to get container status \"9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063\": rpc error: code = NotFound desc = could not find container \"9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063\": container with ID starting with 9a864fd5dad310a3da229ca61bdd97d18c8a9d93f3bdd7804c0ce05c7fc29063 not found: ID does not exist"
Jul 09 18:18:58.470056 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.470032    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"]
Jul 09 18:18:58.474707 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.474687    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5bc9974d7f-qng2q"]
Jul 09 18:18:58.892753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:18:58.892720    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c62c2bed-254e-4bc8-9028-46e495bb3070" path="/var/lib/kubelet/pods/c62c2bed-254e-4bc8-9028-46e495bb3070/volumes"
Jul 09 18:19:05.253272 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:05.253219    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:19:15.253208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:15.253147    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:19:18.075062 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.075033    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"]
Jul 09 18:19:18.075534 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.075301    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62c2bed-254e-4bc8-9028-46e495bb3070" containerName="main"
Jul 09 18:19:18.075534 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.075312    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62c2bed-254e-4bc8-9028-46e495bb3070" containerName="main"
Jul 09 18:19:18.075534 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.075338    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62c2bed-254e-4bc8-9028-46e495bb3070" containerName="storage-initializer"
Jul 09 18:19:18.075534 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.075344    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62c2bed-254e-4bc8-9028-46e495bb3070" containerName="storage-initializer"
Jul 09 18:19:18.075534 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.075404    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c62c2bed-254e-4bc8-9028-46e495bb3070" containerName="main"
Jul 09 18:19:18.131447 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.131410    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"]
Jul 09 18:19:18.131604 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.131516    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.135087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.135061    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 09 18:19:18.206350 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.206277    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.206350 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.206346    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.206564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.206387    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.206564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.206461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.206564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.206530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.206564 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.206554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9js\" (UniqueName: \"kubernetes.io/projected/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-kube-api-access-vm9js\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.307340 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.307284    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.307522 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.307361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9js\" (UniqueName: \"kubernetes.io/projected/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-kube-api-access-vm9js\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.307522 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.307428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.307522 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.307457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.307522 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.307489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.307734 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.307534    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.307789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.307754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.307864 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.307843    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.308006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.307977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.309920 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.309892    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.310246 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.310228    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.318186 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.318159    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9js\" (UniqueName: \"kubernetes.io/projected/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-kube-api-access-vm9js\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.442449 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.442348    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:18.582213 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:18.582158    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"]
Jul 09 18:19:18.585039 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:19:18.585011    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05ec0ea_e454_4169_a736_2c3dfbdd3c72.slice/crio-9292f8621251441453b53e94bef4b3d866b6d06382e5af9c1c001d651767a935 WatchSource:0}: Error finding container 9292f8621251441453b53e94bef4b3d866b6d06382e5af9c1c001d651767a935: Status 404 returned error can't find the container with id 9292f8621251441453b53e94bef4b3d866b6d06382e5af9c1c001d651767a935
Jul 09 18:19:19.535840 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:19.535801    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n" event={"ID":"e05ec0ea-e454-4169-a736-2c3dfbdd3c72","Type":"ContainerStarted","Data":"794fd77e95c193cac656518ca6a8b0cee116d4265d7df05404bb9214ce1c23f2"}
Jul 09 18:19:19.536270 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:19.535900    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n" event={"ID":"e05ec0ea-e454-4169-a736-2c3dfbdd3c72","Type":"ContainerStarted","Data":"9292f8621251441453b53e94bef4b3d866b6d06382e5af9c1c001d651767a935"}
Jul 09 18:19:19.557735 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:19.557689    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n" podStartSLOduration=1.557675207 podStartE2EDuration="1.557675207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:19:19.556385663 +0000 UTC m=+955.202379203" watchObservedRunningTime="2026-07-09 18:19:19.557675207 +0000 UTC m=+955.203668746"
Jul 09 18:19:20.443388 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:20.443346    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:25.252753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:25.252706    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:19:28.442668 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:28.442623    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:28.455453 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:28.455428    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:28.591111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:28.591081    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:19:35.252795 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:35.252752    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:19:45.253211 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:45.253154    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:19:51.525528 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:51.525496    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"]
Jul 09 18:19:55.253010 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:19:55.252964    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:20:05.252958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:05.252913    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:20:06.546025 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.545985    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n" podUID="e05ec0ea-e454-4169-a736-2c3dfbdd3c72" containerName="main" containerID="cri-o://794fd77e95c193cac656518ca6a8b0cee116d4265d7df05404bb9214ce1c23f2" gracePeriod=45
Jul 09 18:20:06.580854 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.580816    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n" podUID="e05ec0ea-e454-4169-a736-2c3dfbdd3c72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/health\": dial tcp 10.133.0.27:8000: connect: connection refused"
Jul 09 18:20:06.755084 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.755048    2558 generic.go:356] "Generic (PLEG): container finished" podID="e05ec0ea-e454-4169-a736-2c3dfbdd3c72" containerID="794fd77e95c193cac656518ca6a8b0cee116d4265d7df05404bb9214ce1c23f2" exitCode=0
Jul 09 18:20:06.755253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.755112    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n" event={"ID":"e05ec0ea-e454-4169-a736-2c3dfbdd3c72","Type":"ContainerDied","Data":"794fd77e95c193cac656518ca6a8b0cee116d4265d7df05404bb9214ce1c23f2"}
Jul 09 18:20:06.809458 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.809432    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:20:06.911889 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.911861    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-model-cache\") pod \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") "
Jul 09 18:20:06.912044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.911905    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-home\") pod \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") "
Jul 09 18:20:06.912044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.911969    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tls-certs\") pod \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") "
Jul 09 18:20:06.912044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.912008    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tmp-dir\") pod \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") "
Jul 09 18:20:06.912197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.912048    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm9js\" (UniqueName: \"kubernetes.io/projected/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-kube-api-access-vm9js\") pod \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") "
Jul 09 18:20:06.912197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.912104    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-dshm\") pod \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\" (UID: \"e05ec0ea-e454-4169-a736-2c3dfbdd3c72\") "
Jul 09 18:20:06.912197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.912141    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-model-cache" (OuterVolumeSpecName: "model-cache") pod "e05ec0ea-e454-4169-a736-2c3dfbdd3c72" (UID: "e05ec0ea-e454-4169-a736-2c3dfbdd3c72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:06.912374 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.912281    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-home" (OuterVolumeSpecName: "home") pod "e05ec0ea-e454-4169-a736-2c3dfbdd3c72" (UID: "e05ec0ea-e454-4169-a736-2c3dfbdd3c72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:06.912374 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.912293    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:20:06.912374 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.912315    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e05ec0ea-e454-4169-a736-2c3dfbdd3c72" (UID: "e05ec0ea-e454-4169-a736-2c3dfbdd3c72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:06.914433 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.914404    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-dshm" (OuterVolumeSpecName: "dshm") pod "e05ec0ea-e454-4169-a736-2c3dfbdd3c72" (UID: "e05ec0ea-e454-4169-a736-2c3dfbdd3c72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:06.914562 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.914457    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e05ec0ea-e454-4169-a736-2c3dfbdd3c72" (UID: "e05ec0ea-e454-4169-a736-2c3dfbdd3c72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:20:06.914562 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:06.914477    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-kube-api-access-vm9js" (OuterVolumeSpecName: "kube-api-access-vm9js") pod "e05ec0ea-e454-4169-a736-2c3dfbdd3c72" (UID: "e05ec0ea-e454-4169-a736-2c3dfbdd3c72"). InnerVolumeSpecName "kube-api-access-vm9js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:20:07.012671 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.012637    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:20:07.012671 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.012667    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:20:07.012671 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.012675    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:20:07.012900 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.012686    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:20:07.012900 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.012694    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm9js\" (UniqueName: \"kubernetes.io/projected/e05ec0ea-e454-4169-a736-2c3dfbdd3c72-kube-api-access-vm9js\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:20:07.760614 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.760578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n" event={"ID":"e05ec0ea-e454-4169-a736-2c3dfbdd3c72","Type":"ContainerDied","Data":"9292f8621251441453b53e94bef4b3d866b6d06382e5af9c1c001d651767a935"}
Jul 09 18:20:07.760614 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.760617    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"
Jul 09 18:20:07.761108 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.760652    2558 scope.go:117] "RemoveContainer" containerID="794fd77e95c193cac656518ca6a8b0cee116d4265d7df05404bb9214ce1c23f2"
Jul 09 18:20:07.783301 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.783272    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"]
Jul 09 18:20:07.788256 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:07.788231    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879b5k2n"]
Jul 09 18:20:08.893697 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:08.893664    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e05ec0ea-e454-4169-a736-2c3dfbdd3c72" path="/var/lib/kubelet/pods/e05ec0ea-e454-4169-a736-2c3dfbdd3c72/volumes"
Jul 09 18:20:15.252947 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:15.252900    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:20:25.252961 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:25.252918    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:20:34.882611 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:34.882572    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"]
Jul 09 18:20:34.883049 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:34.883027    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e05ec0ea-e454-4169-a736-2c3dfbdd3c72" containerName="main"
Jul 09 18:20:34.883092 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:34.883057    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e05ec0ea-e454-4169-a736-2c3dfbdd3c72" containerName="main"
Jul 09 18:20:34.883152 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:34.883140    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e05ec0ea-e454-4169-a736-2c3dfbdd3c72" containerName="main"
Jul 09 18:20:34.889690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:34.889663    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:34.894439 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:34.894415    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 09 18:20:34.896037 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:34.896010    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"]
Jul 09 18:20:35.061529 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.061495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.061529 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.061535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.061745 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.061551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.061745 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.061567    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbz5\" (UniqueName: \"kubernetes.io/projected/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-kube-api-access-npbz5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.061745 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.061589    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.061745 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.061663    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.163126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.163027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.163126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.163078    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.163382 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.163204    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.163382 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.163247    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbz5\" (UniqueName: \"kubernetes.io/projected/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-kube-api-access-npbz5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.163382 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.163279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.163382 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.163373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.163601 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.163569    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.163669 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.163644    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.163870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.163845    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.165536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.165515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.165872 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.165852    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.180003 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.179976    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbz5\" (UniqueName: \"kubernetes.io/projected/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-kube-api-access-npbz5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.200906 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.200878    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:35.253784 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.253706    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:20:35.347738 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.347663    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"]
Jul 09 18:20:35.350235 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:20:35.350207    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03c7e4e_0909_4b05_bb3e_66db9f4b8fb8.slice/crio-4d4265a09b8f331e557a331d876793f2479b5b42b8ee20c6ffe136f57be4c1ed WatchSource:0}: Error finding container 4d4265a09b8f331e557a331d876793f2479b5b42b8ee20c6ffe136f57be4c1ed: Status 404 returned error can't find the container with id 4d4265a09b8f331e557a331d876793f2479b5b42b8ee20c6ffe136f57be4c1ed
Jul 09 18:20:35.888791 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.888745    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx" event={"ID":"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8","Type":"ContainerStarted","Data":"648f91b319f85a02a19fa96032ce99a17b242d843d816f3fd14f849bf4d29fbd"}
Jul 09 18:20:35.889261 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.888810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx" event={"ID":"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8","Type":"ContainerStarted","Data":"4d4265a09b8f331e557a331d876793f2479b5b42b8ee20c6ffe136f57be4c1ed"}
Jul 09 18:20:35.913693 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:35.913626    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx" podStartSLOduration=1.913604384 podStartE2EDuration="1.913604384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:20:35.911363343 +0000 UTC m=+1031.557356881" watchObservedRunningTime="2026-07-09 18:20:35.913604384 +0000 UTC m=+1031.559597925"
Jul 09 18:20:36.201035 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:36.200997    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:45.201911 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:45.201868    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:45.215877 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:45.215851    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:45.262367 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:45.262317    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:20:45.270319 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:45.270295    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:20:45.950975 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:45.950947    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:20:59.696924 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:20:59.696889    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"]
Jul 09 18:21:14.721081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:14.721038    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" containerID="cri-o://8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339" gracePeriod=45
Jul 09 18:21:15.263224 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.263189    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 18:21:15.471083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.471059    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:21:15.613814 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.613719    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-model-cache\") pod \"4d9e9497-5a5d-4363-a506-c7cffed08de0\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") "
Jul 09 18:21:15.613814 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.613768    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9e9497-5a5d-4363-a506-c7cffed08de0-tls-certs\") pod \"4d9e9497-5a5d-4363-a506-c7cffed08de0\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") "
Jul 09 18:21:15.613814 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.613809    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8qnb\" (UniqueName: \"kubernetes.io/projected/4d9e9497-5a5d-4363-a506-c7cffed08de0-kube-api-access-l8qnb\") pod \"4d9e9497-5a5d-4363-a506-c7cffed08de0\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") "
Jul 09 18:21:15.614077 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.613831    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-dshm\") pod \"4d9e9497-5a5d-4363-a506-c7cffed08de0\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") "
Jul 09 18:21:15.614077 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.613853    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-tmp-dir\") pod \"4d9e9497-5a5d-4363-a506-c7cffed08de0\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") "
Jul 09 18:21:15.614077 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.613877    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-kserve-provision-location\") pod \"4d9e9497-5a5d-4363-a506-c7cffed08de0\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") "
Jul 09 18:21:15.614077 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.613898    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-home\") pod \"4d9e9497-5a5d-4363-a506-c7cffed08de0\" (UID: \"4d9e9497-5a5d-4363-a506-c7cffed08de0\") "
Jul 09 18:21:15.614258 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.614072    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d9e9497-5a5d-4363-a506-c7cffed08de0" (UID: "4d9e9497-5a5d-4363-a506-c7cffed08de0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:15.614660 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.614625    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-home" (OuterVolumeSpecName: "home") pod "4d9e9497-5a5d-4363-a506-c7cffed08de0" (UID: "4d9e9497-5a5d-4363-a506-c7cffed08de0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:15.616184 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.616153    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9e9497-5a5d-4363-a506-c7cffed08de0-kube-api-access-l8qnb" (OuterVolumeSpecName: "kube-api-access-l8qnb") pod "4d9e9497-5a5d-4363-a506-c7cffed08de0" (UID: "4d9e9497-5a5d-4363-a506-c7cffed08de0"). InnerVolumeSpecName "kube-api-access-l8qnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:21:15.616294 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.616241    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9e9497-5a5d-4363-a506-c7cffed08de0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d9e9497-5a5d-4363-a506-c7cffed08de0" (UID: "4d9e9497-5a5d-4363-a506-c7cffed08de0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:21:15.616613 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.616592    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-dshm" (OuterVolumeSpecName: "dshm") pod "4d9e9497-5a5d-4363-a506-c7cffed08de0" (UID: "4d9e9497-5a5d-4363-a506-c7cffed08de0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:15.626485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.626428    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d9e9497-5a5d-4363-a506-c7cffed08de0" (UID: "4d9e9497-5a5d-4363-a506-c7cffed08de0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:15.669949 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.669917    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d9e9497-5a5d-4363-a506-c7cffed08de0" (UID: "4d9e9497-5a5d-4363-a506-c7cffed08de0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:15.714582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.714554    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8qnb\" (UniqueName: \"kubernetes.io/projected/4d9e9497-5a5d-4363-a506-c7cffed08de0-kube-api-access-l8qnb\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:21:15.714582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.714580    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:21:15.714744 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.714590    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:21:15.714744 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.714600    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:21:15.714744 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.714608    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:21:15.714744 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.714616    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9e9497-5a5d-4363-a506-c7cffed08de0-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:21:15.714744 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:15.714626    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9e9497-5a5d-4363-a506-c7cffed08de0-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:21:16.064496 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.064450    2558 generic.go:356] "Generic (PLEG): container finished" podID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerID="8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339" exitCode=0
Jul 09 18:21:16.064971 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.064514    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"
Jul 09 18:21:16.064971 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.064512    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" event={"ID":"4d9e9497-5a5d-4363-a506-c7cffed08de0","Type":"ContainerDied","Data":"8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339"}
Jul 09 18:21:16.064971 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.064575    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w" event={"ID":"4d9e9497-5a5d-4363-a506-c7cffed08de0","Type":"ContainerDied","Data":"161f0363311d4b6a82f5975194a5a6681164939ae15dbf65655161f00494b14b"}
Jul 09 18:21:16.064971 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.064601    2558 scope.go:117] "RemoveContainer" containerID="8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339"
Jul 09 18:21:16.073154 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.073140    2558 scope.go:117] "RemoveContainer" containerID="c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16"
Jul 09 18:21:16.080705 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.080688    2558 scope.go:117] "RemoveContainer" containerID="8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339"
Jul 09 18:21:16.080991 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:21:16.080972    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339\": container with ID starting with 8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339 not found: ID does not exist" containerID="8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339"
Jul 09 18:21:16.081039 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.081000    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339"} err="failed to get container status \"8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339\": rpc error: code = NotFound desc = could not find container \"8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339\": container with ID starting with 8e7b91287d8720da67310724c60fd4615606556119b9427fc630e2ece9de8339 not found: ID does not exist"
Jul 09 18:21:16.081039 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.081020    2558 scope.go:117] "RemoveContainer" containerID="c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16"
Jul 09 18:21:16.081230 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:21:16.081215    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16\": container with ID starting with c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16 not found: ID does not exist" containerID="c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16"
Jul 09 18:21:16.081275 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.081234    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16"} err="failed to get container status \"c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16\": rpc error: code = NotFound desc = could not find container \"c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16\": container with ID starting with c3d1fb8b0f89878a8796792a69e3eda375b44a60350dfd280b9fd63ce1ea8c16 not found: ID does not exist"
Jul 09 18:21:16.087680 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.087660    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"]
Jul 09 18:21:16.090890 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.090871    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-865fb4fc77-6mq9w"]
Jul 09 18:21:16.892681 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:16.892638    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" path="/var/lib/kubelet/pods/4d9e9497-5a5d-4363-a506-c7cffed08de0/volumes"
Jul 09 18:21:35.660594 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.660558    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"]
Jul 09 18:21:35.661078 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.660996    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="storage-initializer"
Jul 09 18:21:35.661078 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.661015    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="storage-initializer"
Jul 09 18:21:35.661078 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.661035    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main"
Jul 09 18:21:35.661078 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.661043    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main"
Jul 09 18:21:35.661313 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.661124    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9e9497-5a5d-4363-a506-c7cffed08de0" containerName="main"
Jul 09 18:21:35.665772 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.665749    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.669652 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.669625    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:21:35.669781 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.669736    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 09 18:21:35.680631 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.680604    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"]
Jul 09 18:21:35.780419 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.780369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-dshm\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.780546 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.780418    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tmp-dir\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.780546 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.780491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tls-certs\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.780546 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.780515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.780764 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.780572    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-home\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.780764 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.780610    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6l4t\" (UniqueName: \"kubernetes.io/projected/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kube-api-access-j6l4t\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.780764 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.780682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-model-cache\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tls-certs\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881498 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881498 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881415    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-home\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881498 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6l4t\" (UniqueName: \"kubernetes.io/projected/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kube-api-access-j6l4t\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881497    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-model-cache\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881538    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-dshm\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tmp-dir\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881875 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881847    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881994 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881927    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-home\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881994 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tmp-dir\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.881994 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.881985    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-model-cache\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.884167 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.884136    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-dshm\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.884636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.884615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tls-certs\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.894180 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.894149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6l4t\" (UniqueName: \"kubernetes.io/projected/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kube-api-access-j6l4t\") pod \"auth-invalid-token-test-kserve-6ff9db59f7-w79qc\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:35.976303 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:35.976271    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:36.124192 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:36.120818    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"]
Jul 09 18:21:36.127038 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:21:36.127011    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc753f021_f84b_41c4_bf1d_0fa6499ce4d6.slice/crio-f4bdb4d560c2feb727678c7f6fab2a7686a117e37aead0b84373be3a6536bd6c WatchSource:0}: Error finding container f4bdb4d560c2feb727678c7f6fab2a7686a117e37aead0b84373be3a6536bd6c: Status 404 returned error can't find the container with id f4bdb4d560c2feb727678c7f6fab2a7686a117e37aead0b84373be3a6536bd6c
Jul 09 18:21:36.148870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:36.148846    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" event={"ID":"c753f021-f84b-41c4-bf1d-0fa6499ce4d6","Type":"ContainerStarted","Data":"f4bdb4d560c2feb727678c7f6fab2a7686a117e37aead0b84373be3a6536bd6c"}
Jul 09 18:21:37.154966 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:37.154931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" event={"ID":"c753f021-f84b-41c4-bf1d-0fa6499ce4d6","Type":"ContainerStarted","Data":"f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab"}
Jul 09 18:21:40.170043 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:40.170001    2558 generic.go:356] "Generic (PLEG): container finished" podID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerID="f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab" exitCode=0
Jul 09 18:21:40.170557 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:40.170054    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" event={"ID":"c753f021-f84b-41c4-bf1d-0fa6499ce4d6","Type":"ContainerDied","Data":"f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab"}
Jul 09 18:21:41.175591 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:41.175556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" event={"ID":"c753f021-f84b-41c4-bf1d-0fa6499ce4d6","Type":"ContainerStarted","Data":"7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63"}
Jul 09 18:21:41.202288 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:41.202239    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podStartSLOduration=2.862426663 podStartE2EDuration="6.202219277s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.339792614s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:41.19949952 +0000 UTC m=+1096.845493066" watchObservedRunningTime="2026-07-09 18:21:41.202219277 +0000 UTC m=+1096.848212819"
Jul 09 18:21:41.977041 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:41.977009    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:45.976958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:45.976929    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:21:45.978477 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:45.978447    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:21:55.977731 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:21:55.977679    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:22:05.977283 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:05.977242    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:22:15.976963 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:15.976914    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:22:25.977465 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:25.977418    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:22:35.976834 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:35.976786    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:22:42.333414 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:42.333379    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"]
Jul 09 18:22:45.976897 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:45.976810    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:22:55.976770 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:55.976727    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:22:57.354284 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.354238    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx" podUID="f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" containerName="main" containerID="cri-o://648f91b319f85a02a19fa96032ce99a17b242d843d816f3fd14f849bf4d29fbd" gracePeriod=45
Jul 09 18:22:57.518676 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.518611    2558 generic.go:356] "Generic (PLEG): container finished" podID="f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" containerID="648f91b319f85a02a19fa96032ce99a17b242d843d816f3fd14f849bf4d29fbd" exitCode=0
Jul 09 18:22:57.518780 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.518734    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx" event={"ID":"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8","Type":"ContainerDied","Data":"648f91b319f85a02a19fa96032ce99a17b242d843d816f3fd14f849bf4d29fbd"}
Jul 09 18:22:57.591854 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.591820    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:22:57.691492 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691456    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-model-cache\") pod \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") "
Jul 09 18:22:57.691690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691500    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tmp-dir\") pod \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") "
Jul 09 18:22:57.691690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691547    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-dshm\") pod \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") "
Jul 09 18:22:57.691690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691566    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npbz5\" (UniqueName: \"kubernetes.io/projected/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-kube-api-access-npbz5\") pod \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") "
Jul 09 18:22:57.691690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691593    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tls-certs\") pod \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") "
Jul 09 18:22:57.691690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691623    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-home\") pod \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\" (UID: \"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8\") "
Jul 09 18:22:57.691951 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691752    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" (UID: "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:57.691951 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691849    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" (UID: "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:57.691951 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691881    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:22:57.692104 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.691998    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-home" (OuterVolumeSpecName: "home") pod "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" (UID: "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:57.693974 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.693945    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-kube-api-access-npbz5" (OuterVolumeSpecName: "kube-api-access-npbz5") pod "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" (UID: "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8"). InnerVolumeSpecName "kube-api-access-npbz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:22:57.693974 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.693945    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-dshm" (OuterVolumeSpecName: "dshm") pod "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" (UID: "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:57.694137 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.693980    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" (UID: "f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:22:57.792759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.792704    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:22:57.792759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.792751    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:22:57.792759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.792761    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npbz5\" (UniqueName: \"kubernetes.io/projected/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-kube-api-access-npbz5\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:22:57.792759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.792771    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:22:57.792759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:57.792779    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:22:58.525540 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:58.525502    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx" event={"ID":"f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8","Type":"ContainerDied","Data":"4d4265a09b8f331e557a331d876793f2479b5b42b8ee20c6ffe136f57be4c1ed"}
Jul 09 18:22:58.525540 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:58.525531    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"
Jul 09 18:22:58.526058 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:58.525570    2558 scope.go:117] "RemoveContainer" containerID="648f91b319f85a02a19fa96032ce99a17b242d843d816f3fd14f849bf4d29fbd"
Jul 09 18:22:58.547825 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:58.547797    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"]
Jul 09 18:22:58.551664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:58.551644    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4kqnnx"]
Jul 09 18:22:58.897474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:22:58.897391    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" path="/var/lib/kubelet/pods/f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8/volumes"
Jul 09 18:23:05.977054 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:05.977008    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:23:15.976744 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:15.976698    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:23:21.859892 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.859858    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"]
Jul 09 18:23:21.860371 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.860164    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" containerName="main"
Jul 09 18:23:21.860371 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.860176    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" containerName="main"
Jul 09 18:23:21.860371 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.860235    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03c7e4e-0909-4b05-bb3e-66db9f4b8fb8" containerName="main"
Jul 09 18:23:21.862178 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.862159    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.866371 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.866344    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 09 18:23:21.880141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.880118    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"]
Jul 09 18:23:21.890025 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.889998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.890164 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.890031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.890164 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.890047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72d70e1-cf8a-4758-a6f4-1353135c5631-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.890164 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.890117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.890164 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.890151    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57hq\" (UniqueName: \"kubernetes.io/projected/f72d70e1-cf8a-4758-a6f4-1353135c5631-kube-api-access-p57hq\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.890314 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.890196    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.991571 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.991536    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.991571 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.991574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.991825 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.991604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.991825 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.991626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72d70e1-cf8a-4758-a6f4-1353135c5631-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.991825 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.991691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.991977 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.991828    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57hq\" (UniqueName: \"kubernetes.io/projected/f72d70e1-cf8a-4758-a6f4-1353135c5631-kube-api-access-p57hq\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.992035 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.992002    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.992090 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.992033    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.992144 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.992090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.994076 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.994051    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:21.994490 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:21.994467    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72d70e1-cf8a-4758-a6f4-1353135c5631-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:22.004451 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:22.004419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57hq\" (UniqueName: \"kubernetes.io/projected/f72d70e1-cf8a-4758-a6f4-1353135c5631-kube-api-access-p57hq\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:22.174558 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:22.174467    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:22.319531 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:22.319473    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"]
Jul 09 18:23:22.323164 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:23:22.323140    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72d70e1_cf8a_4758_a6f4_1353135c5631.slice/crio-1a4210c058cf9ca3dcbb363cc5f433488ef9b05649f521660e1b6674474eca09 WatchSource:0}: Error finding container 1a4210c058cf9ca3dcbb363cc5f433488ef9b05649f521660e1b6674474eca09: Status 404 returned error can't find the container with id 1a4210c058cf9ca3dcbb363cc5f433488ef9b05649f521660e1b6674474eca09
Jul 09 18:23:22.635135 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:22.635100    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96" event={"ID":"f72d70e1-cf8a-4758-a6f4-1353135c5631","Type":"ContainerStarted","Data":"1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60"}
Jul 09 18:23:22.635319 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:22.635165    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96" event={"ID":"f72d70e1-cf8a-4758-a6f4-1353135c5631","Type":"ContainerStarted","Data":"1a4210c058cf9ca3dcbb363cc5f433488ef9b05649f521660e1b6674474eca09"}
Jul 09 18:23:22.660503 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:22.660441    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96" podStartSLOduration=1.660416959 podStartE2EDuration="1.660416959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:23:22.659306972 +0000 UTC m=+1198.305300512" watchObservedRunningTime="2026-07-09 18:23:22.660416959 +0000 UTC m=+1198.306410500"
Jul 09 18:23:23.175312 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:23.175274    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:24.839202 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:24.839170    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:23:24.840286 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:24.840250    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:23:25.977762 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:25.977719    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:23:32.175180 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:32.175138    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:32.187594 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:32.187571    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:32.689142 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:32.689114    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:23:35.987168 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:35.987137    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:23:35.994972 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:35.994948    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:23:37.896841 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:37.896808    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"]
Jul 09 18:23:52.919945 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:52.919906    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" containerID="cri-o://7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63" gracePeriod=45
Jul 09 18:23:52.987873 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:52.987832    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 18:23:53.672445 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.672420    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:23:53.763537 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.763507    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tmp-dir\") pod \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") "
Jul 09 18:23:53.763689 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.763564    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-dshm\") pod \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") "
Jul 09 18:23:53.763689 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.763599    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-home\") pod \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") "
Jul 09 18:23:53.763689 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.763623    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6l4t\" (UniqueName: \"kubernetes.io/projected/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kube-api-access-j6l4t\") pod \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") "
Jul 09 18:23:53.763689 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.763677    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kserve-provision-location\") pod \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") "
Jul 09 18:23:53.763867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.763706    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tls-certs\") pod \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") "
Jul 09 18:23:53.763867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.763730    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-model-cache\") pod \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\" (UID: \"c753f021-f84b-41c4-bf1d-0fa6499ce4d6\") "
Jul 09 18:23:53.764186 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.764149    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c753f021-f84b-41c4-bf1d-0fa6499ce4d6" (UID: "c753f021-f84b-41c4-bf1d-0fa6499ce4d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:23:53.764378 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.764360    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-home" (OuterVolumeSpecName: "home") pod "c753f021-f84b-41c4-bf1d-0fa6499ce4d6" (UID: "c753f021-f84b-41c4-bf1d-0fa6499ce4d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:23:53.766271 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.766246    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-dshm" (OuterVolumeSpecName: "dshm") pod "c753f021-f84b-41c4-bf1d-0fa6499ce4d6" (UID: "c753f021-f84b-41c4-bf1d-0fa6499ce4d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:23:53.766610 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.766581    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c753f021-f84b-41c4-bf1d-0fa6499ce4d6" (UID: "c753f021-f84b-41c4-bf1d-0fa6499ce4d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:23:53.766838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.766809    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kube-api-access-j6l4t" (OuterVolumeSpecName: "kube-api-access-j6l4t") pod "c753f021-f84b-41c4-bf1d-0fa6499ce4d6" (UID: "c753f021-f84b-41c4-bf1d-0fa6499ce4d6"). InnerVolumeSpecName "kube-api-access-j6l4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:23:53.769026 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.769000    2558 generic.go:356] "Generic (PLEG): container finished" podID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerID="7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63" exitCode=0
Jul 09 18:23:53.769115 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.769047    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"
Jul 09 18:23:53.769115 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.769079    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" event={"ID":"c753f021-f84b-41c4-bf1d-0fa6499ce4d6","Type":"ContainerDied","Data":"7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63"}
Jul 09 18:23:53.769230 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.769182    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc" event={"ID":"c753f021-f84b-41c4-bf1d-0fa6499ce4d6","Type":"ContainerDied","Data":"f4bdb4d560c2feb727678c7f6fab2a7686a117e37aead0b84373be3a6536bd6c"}
Jul 09 18:23:53.769230 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.769219    2558 scope.go:117] "RemoveContainer" containerID="7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63"
Jul 09 18:23:53.782317 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.782292    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c753f021-f84b-41c4-bf1d-0fa6499ce4d6" (UID: "c753f021-f84b-41c4-bf1d-0fa6499ce4d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:23:53.786049 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.786032    2558 scope.go:117] "RemoveContainer" containerID="f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab"
Jul 09 18:23:53.793904 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.793880    2558 scope.go:117] "RemoveContainer" containerID="7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63"
Jul 09 18:23:53.794184 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:23:53.794161    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63\": container with ID starting with 7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63 not found: ID does not exist" containerID="7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63"
Jul 09 18:23:53.794263 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.794196    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63"} err="failed to get container status \"7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63\": rpc error: code = NotFound desc = could not find container \"7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63\": container with ID starting with 7f93cfec3e67004873b9323c013427e954f12dd43af037b11680565a33bf4d63 not found: ID does not exist"
Jul 09 18:23:53.794263 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.794218    2558 scope.go:117] "RemoveContainer" containerID="f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab"
Jul 09 18:23:53.794476 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:23:53.794456    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab\": container with ID starting with f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab not found: ID does not exist" containerID="f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab"
Jul 09 18:23:53.794551 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.794481    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab"} err="failed to get container status \"f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab\": rpc error: code = NotFound desc = could not find container \"f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab\": container with ID starting with f3536a418a4212ffa264255f4019fdec8d432eaba50d7f4d74cbf85fc90669ab not found: ID does not exist"
Jul 09 18:23:53.830017 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.829990    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c753f021-f84b-41c4-bf1d-0fa6499ce4d6" (UID: "c753f021-f84b-41c4-bf1d-0fa6499ce4d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:23:53.865043 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.865017    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:23:53.865043 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.865040    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6l4t\" (UniqueName: \"kubernetes.io/projected/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kube-api-access-j6l4t\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:23:53.865179 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.865051    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:23:53.865179 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.865061    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:23:53.865179 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.865069    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:23:53.865179 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.865095    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:23:53.865179 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:53.865103    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c753f021-f84b-41c4-bf1d-0fa6499ce4d6-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:23:54.093304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:54.093276    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"]
Jul 09 18:23:54.098108 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:54.098084    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-6ff9db59f7-w79qc"]
Jul 09 18:23:54.892708 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:23:54.892675    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" path="/var/lib/kubelet/pods/c753f021-f84b-41c4-bf1d-0fa6499ce4d6/volumes"
Jul 09 18:24:17.874738 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.874657    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"]
Jul 09 18:24:17.875119 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.875090    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="storage-initializer"
Jul 09 18:24:17.875119 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.875109    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="storage-initializer"
Jul 09 18:24:17.875187 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.875121    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main"
Jul 09 18:24:17.875187 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.875131    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main"
Jul 09 18:24:17.875247 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.875205    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c753f021-f84b-41c4-bf1d-0fa6499ce4d6" containerName="main"
Jul 09 18:24:17.880203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.880178    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:17.883309 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.883288    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:24:17.883309 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.883310    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 09 18:24:17.890371 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.890278    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"]
Jul 09 18:24:17.956948 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.956919    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tmp-dir\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:17.957115 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.956952    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs74d\" (UniqueName: \"kubernetes.io/projected/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kube-api-access-zs74d\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:17.957115 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.956976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-dshm\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:17.957115 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.956995    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-home\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:17.957115 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.957086    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tls-certs\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:17.957278 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.957138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kserve-provision-location\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:17.957278 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:17.957193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-model-cache\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tls-certs\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058045    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kserve-provision-location\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-model-cache\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tmp-dir\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs74d\" (UniqueName: \"kubernetes.io/projected/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kube-api-access-zs74d\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058197    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-dshm\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058470 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058224    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-home\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kserve-provision-location\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-model-cache\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058644 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058579    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-home\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.058726 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.058708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tmp-dir\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.060562 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.060545    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-dshm\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.060800 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.060783    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tls-certs\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.067738 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.067717    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs74d\" (UniqueName: \"kubernetes.io/projected/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kube-api-access-zs74d\") pod \"auth-disabled-test-kserve-5dc95f64cf-qnt8q\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.191219 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.191183    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:18.330670 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.330627    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"]
Jul 09 18:24:18.870184 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.870146    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" event={"ID":"de26eab4-d47e-4f69-82bd-ffd1dd2e5926","Type":"ContainerStarted","Data":"538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11"}
Jul 09 18:24:18.870399 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:18.870248    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" event={"ID":"de26eab4-d47e-4f69-82bd-ffd1dd2e5926","Type":"ContainerStarted","Data":"abb3d47b2dd62ca4679f1b9ebab6fa2a4409076f0d9bab85fa0c1a8ac4877a11"}
Jul 09 18:24:22.895572 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:22.895537    2558 generic.go:356] "Generic (PLEG): container finished" podID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerID="538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11" exitCode=0
Jul 09 18:24:22.895960 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:22.895613    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" event={"ID":"de26eab4-d47e-4f69-82bd-ffd1dd2e5926","Type":"ContainerDied","Data":"538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11"}
Jul 09 18:24:23.902808 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:23.902772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" event={"ID":"de26eab4-d47e-4f69-82bd-ffd1dd2e5926","Type":"ContainerStarted","Data":"9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48"}
Jul 09 18:24:23.933659 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:23.933610    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podStartSLOduration=2.92386779 podStartE2EDuration="6.933592868s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.009725078s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:23.931048017 +0000 UTC m=+1259.577041557" watchObservedRunningTime="2026-07-09 18:24:23.933592868 +0000 UTC m=+1259.579586452"
Jul 09 18:24:24.191609 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:24.191576    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:28.191578 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:28.191537    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:24:28.193301 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:28.193266    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:24:38.191828 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:38.191782    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:24:48.191865 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:48.191816    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:24:52.408458 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:52.408418    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"]
Jul 09 18:24:58.191794 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:24:58.191750    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:25:07.435226 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.435187    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96" podUID="f72d70e1-cf8a-4758-a6f4-1353135c5631" containerName="main" containerID="cri-o://1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60" gracePeriod=45
Jul 09 18:25:07.680216 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.680191    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:25:07.787383 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787320    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-tmp-dir\") pod \"f72d70e1-cf8a-4758-a6f4-1353135c5631\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") "
Jul 09 18:25:07.787383 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787390    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-dshm\") pod \"f72d70e1-cf8a-4758-a6f4-1353135c5631\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") "
Jul 09 18:25:07.787646 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787418    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72d70e1-cf8a-4758-a6f4-1353135c5631-tls-certs\") pod \"f72d70e1-cf8a-4758-a6f4-1353135c5631\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") "
Jul 09 18:25:07.787646 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787441    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-model-cache\") pod \"f72d70e1-cf8a-4758-a6f4-1353135c5631\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") "
Jul 09 18:25:07.787646 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787522    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-home\") pod \"f72d70e1-cf8a-4758-a6f4-1353135c5631\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") "
Jul 09 18:25:07.787646 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787548    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p57hq\" (UniqueName: \"kubernetes.io/projected/f72d70e1-cf8a-4758-a6f4-1353135c5631-kube-api-access-p57hq\") pod \"f72d70e1-cf8a-4758-a6f4-1353135c5631\" (UID: \"f72d70e1-cf8a-4758-a6f4-1353135c5631\") "
Jul 09 18:25:07.787852 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787697    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-model-cache" (OuterVolumeSpecName: "model-cache") pod "f72d70e1-cf8a-4758-a6f4-1353135c5631" (UID: "f72d70e1-cf8a-4758-a6f4-1353135c5631"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:07.787852 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787713    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f72d70e1-cf8a-4758-a6f4-1353135c5631" (UID: "f72d70e1-cf8a-4758-a6f4-1353135c5631"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:07.787955 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787926    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:25:07.787955 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787950    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:25:07.788060 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.787928    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-home" (OuterVolumeSpecName: "home") pod "f72d70e1-cf8a-4758-a6f4-1353135c5631" (UID: "f72d70e1-cf8a-4758-a6f4-1353135c5631"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:07.789693 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.789667    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72d70e1-cf8a-4758-a6f4-1353135c5631-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f72d70e1-cf8a-4758-a6f4-1353135c5631" (UID: "f72d70e1-cf8a-4758-a6f4-1353135c5631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:25:07.790219 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.790195    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-dshm" (OuterVolumeSpecName: "dshm") pod "f72d70e1-cf8a-4758-a6f4-1353135c5631" (UID: "f72d70e1-cf8a-4758-a6f4-1353135c5631"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:07.790308 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.790221    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72d70e1-cf8a-4758-a6f4-1353135c5631-kube-api-access-p57hq" (OuterVolumeSpecName: "kube-api-access-p57hq") pod "f72d70e1-cf8a-4758-a6f4-1353135c5631" (UID: "f72d70e1-cf8a-4758-a6f4-1353135c5631"). InnerVolumeSpecName "kube-api-access-p57hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:25:07.888258 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.888229    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:25:07.888258 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.888255    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57hq\" (UniqueName: \"kubernetes.io/projected/f72d70e1-cf8a-4758-a6f4-1353135c5631-kube-api-access-p57hq\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:25:07.888258 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.888265    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72d70e1-cf8a-4758-a6f4-1353135c5631-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:25:07.888488 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:07.888273    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72d70e1-cf8a-4758-a6f4-1353135c5631-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:25:08.097862 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.097769    2558 generic.go:356] "Generic (PLEG): container finished" podID="f72d70e1-cf8a-4758-a6f4-1353135c5631" containerID="1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60" exitCode=0
Jul 09 18:25:08.097862 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.097819    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"
Jul 09 18:25:08.098081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.097857    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96" event={"ID":"f72d70e1-cf8a-4758-a6f4-1353135c5631","Type":"ContainerDied","Data":"1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60"}
Jul 09 18:25:08.098081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.097923    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96" event={"ID":"f72d70e1-cf8a-4758-a6f4-1353135c5631","Type":"ContainerDied","Data":"1a4210c058cf9ca3dcbb363cc5f433488ef9b05649f521660e1b6674474eca09"}
Jul 09 18:25:08.098081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.097953    2558 scope.go:117] "RemoveContainer" containerID="1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60"
Jul 09 18:25:08.107573 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.107556    2558 scope.go:117] "RemoveContainer" containerID="1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60"
Jul 09 18:25:08.107821 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:25:08.107804    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60\": container with ID starting with 1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60 not found: ID does not exist" containerID="1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60"
Jul 09 18:25:08.107886 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.107828    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60"} err="failed to get container status \"1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60\": rpc error: code = NotFound desc = could not find container \"1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60\": container with ID starting with 1a317e760e2884365838e1ff5751ae8b3ce50e88392b05f164925e7b3253ba60 not found: ID does not exist"
Jul 09 18:25:08.122027 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.122004    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"]
Jul 09 18:25:08.124138 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.124114    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rdv96"]
Jul 09 18:25:08.191907 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.191858    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:25:08.892766 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:08.892729    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72d70e1-cf8a-4758-a6f4-1353135c5631" path="/var/lib/kubelet/pods/f72d70e1-cf8a-4758-a6f4-1353135c5631/volumes"
Jul 09 18:25:18.192121 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:18.192061    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:25:23.780730 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.780695    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"]
Jul 09 18:25:23.781112 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.781095    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72d70e1-cf8a-4758-a6f4-1353135c5631" containerName="main"
Jul 09 18:25:23.781173 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.781117    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72d70e1-cf8a-4758-a6f4-1353135c5631" containerName="main"
Jul 09 18:25:23.781233 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.781220    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72d70e1-cf8a-4758-a6f4-1353135c5631" containerName="main"
Jul 09 18:25:23.783569 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.783549    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:23.787513 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.787486    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:25:23.787639 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.787585    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 09 18:25:23.800378 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.800350    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"]
Jul 09 18:25:23.927210 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.927175    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:23.927395 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.927228    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2d5c72e-60aa-4473-a11d-1c39e8006edf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:23.927395 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.927344    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5g7\" (UniqueName: \"kubernetes.io/projected/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kube-api-access-7b5g7\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:23.927495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.927412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:23.927495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.927448    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:23.927495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.927478    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:23.927605 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.927511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:23.927605 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:23.927548    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.028976 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.028937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2d5c72e-60aa-4473-a11d-1c39e8006edf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.028985    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5g7\" (UniqueName: \"kubernetes.io/projected/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kube-api-access-7b5g7\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029024    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029054    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029078    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029106    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029161    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029469 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029207    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029686 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029593    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029786 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029710    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029786 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029747    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029881 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029802    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.029932 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.029912    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2d5c72e-60aa-4473-a11d-1c39e8006edf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.031715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.031661    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.031919 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.031901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.039126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.039097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5g7\" (UniqueName: \"kubernetes.io/projected/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kube-api-access-7b5g7\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.093794 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.093754    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:24.244096 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:24.244071    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"]
Jul 09 18:25:24.245792 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:25:24.245765    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d5c72e_60aa_4473_a11d_1c39e8006edf.slice/crio-e7462037ed7186b302544707741fb08a49a8e49fa988236503d3451c23b1c862 WatchSource:0}: Error finding container e7462037ed7186b302544707741fb08a49a8e49fa988236503d3451c23b1c862: Status 404 returned error can't find the container with id e7462037ed7186b302544707741fb08a49a8e49fa988236503d3451c23b1c862
Jul 09 18:25:25.181185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:25.181091    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" event={"ID":"d2d5c72e-60aa-4473-a11d-1c39e8006edf","Type":"ContainerStarted","Data":"548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f"}
Jul 09 18:25:25.181185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:25.181162    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" event={"ID":"d2d5c72e-60aa-4473-a11d-1c39e8006edf","Type":"ContainerStarted","Data":"e7462037ed7186b302544707741fb08a49a8e49fa988236503d3451c23b1c862"}
Jul 09 18:25:28.192534 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:28.192490    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:25:29.203251 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:29.203211    2558 generic.go:356] "Generic (PLEG): container finished" podID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerID="548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f" exitCode=0
Jul 09 18:25:29.203839 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:29.203284    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" event={"ID":"d2d5c72e-60aa-4473-a11d-1c39e8006edf","Type":"ContainerDied","Data":"548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f"}
Jul 09 18:25:30.211059 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:30.211015    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" event={"ID":"d2d5c72e-60aa-4473-a11d-1c39e8006edf","Type":"ContainerStarted","Data":"391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a"}
Jul 09 18:25:30.242706 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:30.242649    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podStartSLOduration=3.092568865 podStartE2EDuration="7.242631551s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.150062686s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:25:30.24013584 +0000 UTC m=+1325.886129379" watchObservedRunningTime="2026-07-09 18:25:30.242631551 +0000 UTC m=+1325.888625090"
Jul 09 18:25:31.094133 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:31.094090    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:34.094167 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:34.094133    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:25:34.096193 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:34.096138    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 09 18:25:38.192450 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:38.192400    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:25:44.094615 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:44.094570    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 09 18:25:48.192397 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:48.192281    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:25:54.094952 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:54.094907    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 09 18:25:58.192570 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:25:58.192515    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:26:04.095021 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:04.094971    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 09 18:26:08.191785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:08.191730    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Jul 09 18:26:14.094690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:14.094640    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 09 18:26:18.202270 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:18.202232    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:26:18.210261 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:18.210233    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:26:23.382113 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:23.382069    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"]
Jul 09 18:26:24.094676 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:24.094626    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 09 18:26:34.094189 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:34.094151    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 09 18:26:38.407492 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:38.407427    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main" containerID="cri-o://9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48" gracePeriod=45
Jul 09 18:26:39.087437 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.087408    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:26:39.205118 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205089    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-model-cache\") pod \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") "
Jul 09 18:26:39.205295 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205137    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tmp-dir\") pod \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") "
Jul 09 18:26:39.205295 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205188    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tls-certs\") pod \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") "
Jul 09 18:26:39.205295 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205211    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kserve-provision-location\") pod \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") "
Jul 09 18:26:39.205295 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205234    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-dshm\") pod \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") "
Jul 09 18:26:39.205295 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205258    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-home\") pod \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") "
Jul 09 18:26:39.205600 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205301    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs74d\" (UniqueName: \"kubernetes.io/projected/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kube-api-access-zs74d\") pod \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\" (UID: \"de26eab4-d47e-4f69-82bd-ffd1dd2e5926\") "
Jul 09 18:26:39.205600 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205395    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-model-cache" (OuterVolumeSpecName: "model-cache") pod "de26eab4-d47e-4f69-82bd-ffd1dd2e5926" (UID: "de26eab4-d47e-4f69-82bd-ffd1dd2e5926"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:39.205600 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205595    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:26:39.206016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.205993    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-home" (OuterVolumeSpecName: "home") pod "de26eab4-d47e-4f69-82bd-ffd1dd2e5926" (UID: "de26eab4-d47e-4f69-82bd-ffd1dd2e5926"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:39.207602 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.207574    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de26eab4-d47e-4f69-82bd-ffd1dd2e5926" (UID: "de26eab4-d47e-4f69-82bd-ffd1dd2e5926"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:39.207833 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.207800    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kube-api-access-zs74d" (OuterVolumeSpecName: "kube-api-access-zs74d") pod "de26eab4-d47e-4f69-82bd-ffd1dd2e5926" (UID: "de26eab4-d47e-4f69-82bd-ffd1dd2e5926"). InnerVolumeSpecName "kube-api-access-zs74d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:39.207962 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.207940    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-dshm" (OuterVolumeSpecName: "dshm") pod "de26eab4-d47e-4f69-82bd-ffd1dd2e5926" (UID: "de26eab4-d47e-4f69-82bd-ffd1dd2e5926"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:39.217976 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.217950    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de26eab4-d47e-4f69-82bd-ffd1dd2e5926" (UID: "de26eab4-d47e-4f69-82bd-ffd1dd2e5926"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:39.280863 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.280817    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de26eab4-d47e-4f69-82bd-ffd1dd2e5926" (UID: "de26eab4-d47e-4f69-82bd-ffd1dd2e5926"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:39.306337 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.306301    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs74d\" (UniqueName: \"kubernetes.io/projected/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kube-api-access-zs74d\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:26:39.306474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.306352    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:26:39.306474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.306366    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:26:39.306474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.306375    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:26:39.306474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.306382    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:26:39.306474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.306391    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de26eab4-d47e-4f69-82bd-ffd1dd2e5926-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:26:39.551649 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.551556    2558 generic.go:356] "Generic (PLEG): container finished" podID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerID="9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48" exitCode=0
Jul 09 18:26:39.551649 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.551621    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"
Jul 09 18:26:39.551649 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.551631    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" event={"ID":"de26eab4-d47e-4f69-82bd-ffd1dd2e5926","Type":"ContainerDied","Data":"9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48"}
Jul 09 18:26:39.552179 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.551684    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q" event={"ID":"de26eab4-d47e-4f69-82bd-ffd1dd2e5926","Type":"ContainerDied","Data":"abb3d47b2dd62ca4679f1b9ebab6fa2a4409076f0d9bab85fa0c1a8ac4877a11"}
Jul 09 18:26:39.552179 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.551708    2558 scope.go:117] "RemoveContainer" containerID="9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48"
Jul 09 18:26:39.561306 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.561284    2558 scope.go:117] "RemoveContainer" containerID="538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11"
Jul 09 18:26:39.570718 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.570699    2558 scope.go:117] "RemoveContainer" containerID="9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48"
Jul 09 18:26:39.571038 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:26:39.571017    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48\": container with ID starting with 9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48 not found: ID does not exist" containerID="9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48"
Jul 09 18:26:39.571120 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.571051    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48"} err="failed to get container status \"9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48\": rpc error: code = NotFound desc = could not find container \"9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48\": container with ID starting with 9f0a9c684df8b2d6a6ecfe35d4c506cffd380e8a60b0f075536f31fcb3de2a48 not found: ID does not exist"
Jul 09 18:26:39.571120 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.571075    2558 scope.go:117] "RemoveContainer" containerID="538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11"
Jul 09 18:26:39.571397 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:26:39.571372    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11\": container with ID starting with 538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11 not found: ID does not exist" containerID="538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11"
Jul 09 18:26:39.571446 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.571409    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11"} err="failed to get container status \"538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11\": rpc error: code = NotFound desc = could not find container \"538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11\": container with ID starting with 538f95d504989ebd4fa919478ad33039a0f675fbf744e4bd09e10b2d63de5c11 not found: ID does not exist"
Jul 09 18:26:39.579013 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.578984    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"]
Jul 09 18:26:39.584034 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:39.584000    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5dc95f64cf-qnt8q"]
Jul 09 18:26:40.895258 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:40.895219    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" path="/var/lib/kubelet/pods/de26eab4-d47e-4f69-82bd-ffd1dd2e5926/volumes"
Jul 09 18:26:44.094681 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:44.094630    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 09 18:26:54.104746 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:54.104717    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:26:54.112607 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:26:54.112577    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:27:08.676196 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:08.676158    2558 generic.go:356] "Generic (PLEG): container finished" podID="35bb2f54-cb55-4392-8f28-8b2709b69211" containerID="305104d3d71408425e4382d8d6f24bfddbe98330afb2c8c820bbc364ee2f3436" exitCode=0
Jul 09 18:27:08.676752 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:08.676213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" event={"ID":"35bb2f54-cb55-4392-8f28-8b2709b69211","Type":"ContainerDied","Data":"305104d3d71408425e4382d8d6f24bfddbe98330afb2c8c820bbc364ee2f3436"}
Jul 09 18:27:08.676752 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:08.676683    2558 scope.go:117] "RemoveContainer" containerID="305104d3d71408425e4382d8d6f24bfddbe98330afb2c8c820bbc364ee2f3436"
Jul 09 18:27:08.677704 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:08.677670    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 18:27:08.677810 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:08.677746    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 18:27:08.677810 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:08.677780    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 18:27:09.685701 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:09.685662    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" event={"ID":"35bb2f54-cb55-4392-8f28-8b2709b69211","Type":"ContainerStarted","Data":"dee8b7f688ff2f05a6a4c8c2c4c163f30a0b32cb12daffa23903ded26bc0d8cd"}
Jul 09 18:27:09.971155 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:09.971127    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:27:09.971362 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:09.971305    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" podUID="35bb2f54-cb55-4392-8f28-8b2709b69211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Jul 09 18:27:10.972131 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:10.972075    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" podUID="35bb2f54-cb55-4392-8f28-8b2709b69211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Jul 09 18:27:11.971356 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:11.971287    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" podUID="35bb2f54-cb55-4392-8f28-8b2709b69211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Jul 09 18:27:12.971336 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:12.971281    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" podUID="35bb2f54-cb55-4392-8f28-8b2709b69211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Jul 09 18:27:13.971944 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:13.971900    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" podUID="35bb2f54-cb55-4392-8f28-8b2709b69211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Jul 09 18:27:14.971691 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:14.971646    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5" podUID="35bb2f54-cb55-4392-8f28-8b2709b69211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Jul 09 18:27:15.975982 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:15.975903    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:27:15.976396 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:15.976256    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:27:15.976969 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:15.976944    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-755f5"
Jul 09 18:27:26.998759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:26.998728    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"]
Jul 09 18:27:42.023592 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.023540    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" containerID="cri-o://391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a" gracePeriod=45
Jul 09 18:27:42.105536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.105493    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 09 18:27:42.678434 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.678406    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:27:42.765240 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765143    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-model-cache\") pod \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") "
Jul 09 18:27:42.765240 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765190    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-home\") pod \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") "
Jul 09 18:27:42.765240 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765216    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tls-certs\") pod \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") "
Jul 09 18:27:42.765565 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765278    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2d5c72e-60aa-4473-a11d-1c39e8006edf-cabundle-cert\") pod \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") "
Jul 09 18:27:42.765565 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765297    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tmp-dir\") pod \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") "
Jul 09 18:27:42.765565 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765365    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kserve-provision-location\") pod \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") "
Jul 09 18:27:42.765565 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765393    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-dshm\") pod \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") "
Jul 09 18:27:42.765565 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765418    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b5g7\" (UniqueName: \"kubernetes.io/projected/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kube-api-access-7b5g7\") pod \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\" (UID: \"d2d5c72e-60aa-4473-a11d-1c39e8006edf\") "
Jul 09 18:27:42.765565 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765538    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2d5c72e-60aa-4473-a11d-1c39e8006edf" (UID: "d2d5c72e-60aa-4473-a11d-1c39e8006edf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:42.765846 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765640    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:27:42.765846 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765658    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-home" (OuterVolumeSpecName: "home") pod "d2d5c72e-60aa-4473-a11d-1c39e8006edf" (UID: "d2d5c72e-60aa-4473-a11d-1c39e8006edf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:42.765846 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.765755    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d5c72e-60aa-4473-a11d-1c39e8006edf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2d5c72e-60aa-4473-a11d-1c39e8006edf" (UID: "d2d5c72e-60aa-4473-a11d-1c39e8006edf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:27:42.767715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.767690    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2d5c72e-60aa-4473-a11d-1c39e8006edf" (UID: "d2d5c72e-60aa-4473-a11d-1c39e8006edf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:27:42.767715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.767700    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kube-api-access-7b5g7" (OuterVolumeSpecName: "kube-api-access-7b5g7") pod "d2d5c72e-60aa-4473-a11d-1c39e8006edf" (UID: "d2d5c72e-60aa-4473-a11d-1c39e8006edf"). InnerVolumeSpecName "kube-api-access-7b5g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:27:42.767858 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.767742    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-dshm" (OuterVolumeSpecName: "dshm") pod "d2d5c72e-60aa-4473-a11d-1c39e8006edf" (UID: "d2d5c72e-60aa-4473-a11d-1c39e8006edf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:42.785919 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.785857    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2d5c72e-60aa-4473-a11d-1c39e8006edf" (UID: "d2d5c72e-60aa-4473-a11d-1c39e8006edf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:42.829634 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.829592    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2d5c72e-60aa-4473-a11d-1c39e8006edf" (UID: "d2d5c72e-60aa-4473-a11d-1c39e8006edf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:27:42.835416 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.835380    2558 generic.go:356] "Generic (PLEG): container finished" podID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerID="391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a" exitCode=0
Jul 09 18:27:42.835556 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.835430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" event={"ID":"d2d5c72e-60aa-4473-a11d-1c39e8006edf","Type":"ContainerDied","Data":"391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a"}
Jul 09 18:27:42.835556 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.835500    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj" event={"ID":"d2d5c72e-60aa-4473-a11d-1c39e8006edf","Type":"ContainerDied","Data":"e7462037ed7186b302544707741fb08a49a8e49fa988236503d3451c23b1c862"}
Jul 09 18:27:42.835556 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.835441    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"
Jul 09 18:27:42.835657 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.835523    2558 scope.go:117] "RemoveContainer" containerID="391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a"
Jul 09 18:27:42.845288 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.845254    2558 scope.go:117] "RemoveContainer" containerID="548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f"
Jul 09 18:27:42.853611 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.853590    2558 scope.go:117] "RemoveContainer" containerID="391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a"
Jul 09 18:27:42.853867 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:27:42.853846    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a\": container with ID starting with 391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a not found: ID does not exist" containerID="391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a"
Jul 09 18:27:42.853911 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.853877    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a"} err="failed to get container status \"391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a\": rpc error: code = NotFound desc = could not find container \"391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a\": container with ID starting with 391d34c40dc86a15dc3bc06ff74e48c59f40952a84235d9cc398234ba2d9719a not found: ID does not exist"
Jul 09 18:27:42.853911 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.853894    2558 scope.go:117] "RemoveContainer" containerID="548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f"
Jul 09 18:27:42.854144 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:27:42.854127    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f\": container with ID starting with 548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f not found: ID does not exist" containerID="548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f"
Jul 09 18:27:42.854185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.854151    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f"} err="failed to get container status \"548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f\": rpc error: code = NotFound desc = could not find container \"548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f\": container with ID starting with 548330e37c3fc3c583ca16a1eb677178238aee4639075b2e7d86b8237aa3040f not found: ID does not exist"
Jul 09 18:27:42.861532 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.861504    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"]
Jul 09 18:27:42.866471 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.866440    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:27:42.866609 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.866475    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2d5c72e-60aa-4473-a11d-1c39e8006edf-cabundle-cert\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:27:42.866609 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.866510    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:27:42.866609 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.866569    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:27:42.866609 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.866584    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:27:42.866750 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.866611    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b5g7\" (UniqueName: \"kubernetes.io/projected/d2d5c72e-60aa-4473-a11d-1c39e8006edf-kube-api-access-7b5g7\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:27:42.866820 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.866799    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d5c72e-60aa-4473-a11d-1c39e8006edf-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:27:42.867782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.867763    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-5794fffdfb-vx8vj"]
Jul 09 18:27:42.892135 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:27:42.892101    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" path="/var/lib/kubelet/pods/d2d5c72e-60aa-4473-a11d-1c39e8006edf/volumes"
Jul 09 18:28:02.084996 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.084963    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"]
Jul 09 18:28:02.085580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085410    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main"
Jul 09 18:28:02.085580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085429    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main"
Jul 09 18:28:02.085580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085451    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="storage-initializer"
Jul 09 18:28:02.085580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085460    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="storage-initializer"
Jul 09 18:28:02.085580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085474    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main"
Jul 09 18:28:02.085580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085483    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main"
Jul 09 18:28:02.085580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085509    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="storage-initializer"
Jul 09 18:28:02.085580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085517    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="storage-initializer"
Jul 09 18:28:02.085962 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085586    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d5c72e-60aa-4473-a11d-1c39e8006edf" containerName="main"
Jul 09 18:28:02.085962 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.085598    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="de26eab4-d47e-4f69-82bd-ffd1dd2e5926" containerName="main"
Jul 09 18:28:02.088626 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.088605    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.091957 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.091935    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:28:02.092081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.091938    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:28:02.092081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.092007    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:28:02.096240 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.096214    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 09 18:28:02.107158 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.107137    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"]
Jul 09 18:28:02.229387 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.229346    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.229590 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.229399    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.229590 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.229439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtlsv\" (UniqueName: \"kubernetes.io/projected/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kube-api-access-qtlsv\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.229590 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.229469    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.229590 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.229494    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.229590 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.229521    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.229590 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.229555    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.229590 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.229576    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.330767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.330734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.330767 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.330768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331008 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.330826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331008 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.330859    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331008 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.330900    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtlsv\" (UniqueName: \"kubernetes.io/projected/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kube-api-access-qtlsv\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331008 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.330931    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331008 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.330953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331008 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.330978    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331309 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.331223    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331309 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.331264    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331309 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.331285    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331501 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.331410    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.331701 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.331645    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.333472 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.333449    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.333620 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.333602    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.348289 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.348229    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtlsv\" (UniqueName: \"kubernetes.io/projected/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kube-api-access-qtlsv\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.398886 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.398857    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:02.544658 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.544540    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"]
Jul 09 18:28:02.547517 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:28:02.547490    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4366bc9_4fc1_4dc0_8a1b_b892523a599c.slice/crio-ca078d035d85b9d9d07185b9133ea24903ddb2f078907803177ebf7145ee9320 WatchSource:0}: Error finding container ca078d035d85b9d9d07185b9133ea24903ddb2f078907803177ebf7145ee9320: Status 404 returned error can't find the container with id ca078d035d85b9d9d07185b9133ea24903ddb2f078907803177ebf7145ee9320
Jul 09 18:28:02.915024 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.914931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" event={"ID":"c4366bc9-4fc1-4dc0-8a1b-b892523a599c","Type":"ContainerStarted","Data":"a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9"}
Jul 09 18:28:02.915024 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:02.914996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" event={"ID":"c4366bc9-4fc1-4dc0-8a1b-b892523a599c","Type":"ContainerStarted","Data":"ca078d035d85b9d9d07185b9133ea24903ddb2f078907803177ebf7145ee9320"}
Jul 09 18:28:06.937556 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:06.937524    2558 generic.go:356] "Generic (PLEG): container finished" podID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerID="a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9" exitCode=0
Jul 09 18:28:06.937985 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:06.937592    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" event={"ID":"c4366bc9-4fc1-4dc0-8a1b-b892523a599c","Type":"ContainerDied","Data":"a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9"}
Jul 09 18:28:07.943390 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:07.943356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" event={"ID":"c4366bc9-4fc1-4dc0-8a1b-b892523a599c","Type":"ContainerStarted","Data":"2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb"}
Jul 09 18:28:07.980817 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:07.980755    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podStartSLOduration=1.724164067 podStartE2EDuration="5.980732726s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.256568659s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:07.98054419 +0000 UTC m=+1483.626537728" watchObservedRunningTime="2026-07-09 18:28:07.980732726 +0000 UTC m=+1483.626726272"
Jul 09 18:28:08.399277 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:08.399186    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:12.399623 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:12.399582    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:28:12.400944 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:12.400915    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 18:28:22.400210 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:22.400169    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 18:28:24.863566 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:24.863529    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:28:24.866734 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:24.866712    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:28:32.399725 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:32.399676    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 18:28:41.006733 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.006695    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"]
Jul 09 18:28:41.013042 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.013010    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.017429 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.017409    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 09 18:28:41.029945 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.029910    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"]
Jul 09 18:28:41.186808 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.186777    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.186993 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.186814    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.186993 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.186839    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hkk\" (UniqueName: \"kubernetes.io/projected/a96c46fa-c37e-4ea0-883e-63bc6c561974-kube-api-access-t2hkk\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.186993 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.186962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.187117 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.187018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96c46fa-c37e-4ea0-883e-63bc6c561974-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.187117 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.187056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.187117 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.187085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.288658 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.288551    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.288658 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.288609    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.288658 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.288643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hkk\" (UniqueName: \"kubernetes.io/projected/a96c46fa-c37e-4ea0-883e-63bc6c561974-kube-api-access-t2hkk\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.288974 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.288705    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.288974 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.288741    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96c46fa-c37e-4ea0-883e-63bc6c561974-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.288974 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.288788    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.288974 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.288816    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.289278 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.289251    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.289733 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.289711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.289876 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.289853    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.289987 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.289708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.292031 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.291998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.292135 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.292087    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96c46fa-c37e-4ea0-883e-63bc6c561974-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.310600 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.310563    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hkk\" (UniqueName: \"kubernetes.io/projected/a96c46fa-c37e-4ea0-883e-63bc6c561974-kube-api-access-t2hkk\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.324503 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.324469    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:41.492986 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:41.492939    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"]
Jul 09 18:28:41.494708 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:28:41.494674    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96c46fa_c37e_4ea0_883e_63bc6c561974.slice/crio-54416ef43bfc8f0e02946e6f9e1e3cd3e2343f1a5772d6294803de4e2219707a WatchSource:0}: Error finding container 54416ef43bfc8f0e02946e6f9e1e3cd3e2343f1a5772d6294803de4e2219707a: Status 404 returned error can't find the container with id 54416ef43bfc8f0e02946e6f9e1e3cd3e2343f1a5772d6294803de4e2219707a
Jul 09 18:28:42.096319 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:42.096283    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" event={"ID":"a96c46fa-c37e-4ea0-883e-63bc6c561974","Type":"ContainerStarted","Data":"178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a"}
Jul 09 18:28:42.096789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:42.096363    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" event={"ID":"a96c46fa-c37e-4ea0-883e-63bc6c561974","Type":"ContainerStarted","Data":"54416ef43bfc8f0e02946e6f9e1e3cd3e2343f1a5772d6294803de4e2219707a"}
Jul 09 18:28:42.400213 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:42.400122    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 18:28:45.113673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:45.113621    2558 generic.go:356] "Generic (PLEG): container finished" podID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerID="178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a" exitCode=0
Jul 09 18:28:45.114172 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:45.113687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" event={"ID":"a96c46fa-c37e-4ea0-883e-63bc6c561974","Type":"ContainerDied","Data":"178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a"}
Jul 09 18:28:46.120071 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:46.119966    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" event={"ID":"a96c46fa-c37e-4ea0-883e-63bc6c561974","Type":"ContainerStarted","Data":"6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b"}
Jul 09 18:28:46.149399 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:46.149320    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podStartSLOduration=2.618770112 podStartE2EDuration="6.149304663s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.530534551s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:46.1464025 +0000 UTC m=+1521.792396223" watchObservedRunningTime="2026-07-09 18:28:46.149304663 +0000 UTC m=+1521.795298202"
Jul 09 18:28:46.325060 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:46.325016    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:51.325235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:51.325193    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:28:51.326939 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:51.326910    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:28:52.399802 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:28:52.399758    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 18:29:01.325012 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:01.324963    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:29:02.399604 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:02.399560    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 18:29:11.325437 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:11.325384    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:29:12.399768 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:12.399714    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 18:29:21.325660 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:21.325612    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:29:22.400023 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:22.399966    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 18:29:31.325060 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:31.325019    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:29:32.409513 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:32.409480    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:29:32.417496 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:32.417466    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:29:33.754486 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:33.754451    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"]
Jul 09 18:29:41.325178 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:41.325113    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:29:49.384735 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:49.384675    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main" containerID="cri-o://2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb" gracePeriod=45
Jul 09 18:29:50.025252 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.025223    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:29:50.087724 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.087645    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-model-cache\") pod \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") "
Jul 09 18:29:50.087724 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.087683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kserve-provision-location\") pod \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") "
Jul 09 18:29:50.087724 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.087708    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-cabundle-cert\") pod \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") "
Jul 09 18:29:50.087980 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.087735    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-dshm\") pod \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") "
Jul 09 18:29:50.087980 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.087763    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtlsv\" (UniqueName: \"kubernetes.io/projected/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kube-api-access-qtlsv\") pod \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") "
Jul 09 18:29:50.087980 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.087814    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tmp-dir\") pod \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") "
Jul 09 18:29:50.087980 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.087848    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tls-certs\") pod \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") "
Jul 09 18:29:50.087980 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.087874    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-home\") pod \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\" (UID: \"c4366bc9-4fc1-4dc0-8a1b-b892523a599c\") "
Jul 09 18:29:50.087980 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.087967    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4366bc9-4fc1-4dc0-8a1b-b892523a599c" (UID: "c4366bc9-4fc1-4dc0-8a1b-b892523a599c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:29:50.088253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.088027    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4366bc9-4fc1-4dc0-8a1b-b892523a599c" (UID: "c4366bc9-4fc1-4dc0-8a1b-b892523a599c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:29:50.088253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.088173    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:29:50.088253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.088193    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-cabundle-cert\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:29:50.088437 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.088265    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4366bc9-4fc1-4dc0-8a1b-b892523a599c" (UID: "c4366bc9-4fc1-4dc0-8a1b-b892523a599c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:29:50.088437 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.088342    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-home" (OuterVolumeSpecName: "home") pod "c4366bc9-4fc1-4dc0-8a1b-b892523a599c" (UID: "c4366bc9-4fc1-4dc0-8a1b-b892523a599c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:29:50.090750 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.090708    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kube-api-access-qtlsv" (OuterVolumeSpecName: "kube-api-access-qtlsv") pod "c4366bc9-4fc1-4dc0-8a1b-b892523a599c" (UID: "c4366bc9-4fc1-4dc0-8a1b-b892523a599c"). InnerVolumeSpecName "kube-api-access-qtlsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:29:50.090869 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.090810    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4366bc9-4fc1-4dc0-8a1b-b892523a599c" (UID: "c4366bc9-4fc1-4dc0-8a1b-b892523a599c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:29:50.090869 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.090814    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-dshm" (OuterVolumeSpecName: "dshm") pod "c4366bc9-4fc1-4dc0-8a1b-b892523a599c" (UID: "c4366bc9-4fc1-4dc0-8a1b-b892523a599c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:29:50.152940 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.152908    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4366bc9-4fc1-4dc0-8a1b-b892523a599c" (UID: "c4366bc9-4fc1-4dc0-8a1b-b892523a599c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:29:50.188821 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.188793    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:29:50.188821 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.188822    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:29:50.188993 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.188838    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtlsv\" (UniqueName: \"kubernetes.io/projected/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-kube-api-access-qtlsv\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:29:50.188993 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.188850    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:29:50.188993 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.188862    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:29:50.188993 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.188871    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4366bc9-4fc1-4dc0-8a1b-b892523a599c-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:29:50.432674 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.432585    2558 generic.go:356] "Generic (PLEG): container finished" podID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerID="2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb" exitCode=0
Jul 09 18:29:50.432674 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.432627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" event={"ID":"c4366bc9-4fc1-4dc0-8a1b-b892523a599c","Type":"ContainerDied","Data":"2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb"}
Jul 09 18:29:50.432674 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.432670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29" event={"ID":"c4366bc9-4fc1-4dc0-8a1b-b892523a599c","Type":"ContainerDied","Data":"ca078d035d85b9d9d07185b9133ea24903ddb2f078907803177ebf7145ee9320"}
Jul 09 18:29:50.433126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.432697    2558 scope.go:117] "RemoveContainer" containerID="2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb"
Jul 09 18:29:50.433126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.432696    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"
Jul 09 18:29:50.443753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.443735    2558 scope.go:117] "RemoveContainer" containerID="a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9"
Jul 09 18:29:50.452043 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.452025    2558 scope.go:117] "RemoveContainer" containerID="2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb"
Jul 09 18:29:50.452295 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:29:50.452272    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb\": container with ID starting with 2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb not found: ID does not exist" containerID="2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb"
Jul 09 18:29:50.452369 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.452305    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb"} err="failed to get container status \"2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb\": rpc error: code = NotFound desc = could not find container \"2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb\": container with ID starting with 2d09cbed56f890c587baf80451eeb327c8f3ebfda7f2e9fe8643fa95ffb23aeb not found: ID does not exist"
Jul 09 18:29:50.452369 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.452347    2558 scope.go:117] "RemoveContainer" containerID="a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9"
Jul 09 18:29:50.452597 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:29:50.452578    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9\": container with ID starting with a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9 not found: ID does not exist" containerID="a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9"
Jul 09 18:29:50.452637 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.452604    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9"} err="failed to get container status \"a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9\": rpc error: code = NotFound desc = could not find container \"a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9\": container with ID starting with a9274705f3543ab696392bf7981e26bd7797682beaf925766362e8bcebd1bdd9 not found: ID does not exist"
Jul 09 18:29:50.461100 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.461078    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"]
Jul 09 18:29:50.468782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.468760    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9765576c5-lnr29"]
Jul 09 18:29:50.893033 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:50.893001    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" path="/var/lib/kubelet/pods/c4366bc9-4fc1-4dc0-8a1b-b892523a599c/volumes"
Jul 09 18:29:51.325882 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:29:51.325832    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:30:01.325112 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:01.325065    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:30:11.325175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:11.325118    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:30:21.325234 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:21.325134    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:30:25.089243 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.089209    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"]
Jul 09 18:30:25.089669 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.089526    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="storage-initializer"
Jul 09 18:30:25.089669 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.089539    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="storage-initializer"
Jul 09 18:30:25.089669 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.089547    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main"
Jul 09 18:30:25.089669 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.089553    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main"
Jul 09 18:30:25.089669 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.089618    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4366bc9-4fc1-4dc0-8a1b-b892523a599c" containerName="main"
Jul 09 18:30:25.094902 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.094873    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.098200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.098176    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 09 18:30:25.111514 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.111483    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"]
Jul 09 18:30:25.188644 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.188609    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.188644 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.188645    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.188898 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.188674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.188898 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.188778    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.188898 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.188869    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.289785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.289688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.289785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.289735    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk2kb\" (UniqueName: \"kubernetes.io/projected/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-kube-api-access-jk2kb\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.289785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.289774    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.290065 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.289825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.290065 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.289853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.290065 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.289879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.290253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.290228    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.290303 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.290262    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.290303 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.290279    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.292154 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.292129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.292534 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.292514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.391010 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.390973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk2kb\" (UniqueName: \"kubernetes.io/projected/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-kube-api-access-jk2kb\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:25.401496 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:25.401467    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk2kb\" (UniqueName: \"kubernetes.io/projected/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-kube-api-access-jk2kb\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:30.028683 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:30.028637    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:30.036526 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:30.036492    2558 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:30.129657 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:30.129617    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:30.129826 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:30.129770    2558 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:30.133576 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:30.133548    2558 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:30.263313 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:30.263269    2558 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d55ac04b2c83230ef464e6ac64fdc05efc871de6bd51fec133c9f911cb36e9ef/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:30.282071 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:30.281989    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:30.505385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:30.505349    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:30.653996 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:30.653964    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"]
Jul 09 18:30:30.658279 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:30:30.658245    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3ef116_7b8e_40a7_bf85_d97dc7225fe4.slice/crio-7b32962cd13d5f0246d7e4d75f695f532cb85512ecef37afa4c44e2e6e0d7d23 WatchSource:0}: Error finding container 7b32962cd13d5f0246d7e4d75f695f532cb85512ecef37afa4c44e2e6e0d7d23: Status 404 returned error can't find the container with id 7b32962cd13d5f0246d7e4d75f695f532cb85512ecef37afa4c44e2e6e0d7d23
Jul 09 18:30:31.325229 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:31.325176    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:30:31.614870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:31.614747    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" event={"ID":"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4","Type":"ContainerStarted","Data":"828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb"}
Jul 09 18:30:31.614870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:31.614822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" event={"ID":"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4","Type":"ContainerStarted","Data":"7b32962cd13d5f0246d7e4d75f695f532cb85512ecef37afa4c44e2e6e0d7d23"}
Jul 09 18:30:31.640773 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:31.640714    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podStartSLOduration=6.640697883 podStartE2EDuration="6.640697883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:31.639376386 +0000 UTC m=+1627.285369927" watchObservedRunningTime="2026-07-09 18:30:31.640697883 +0000 UTC m=+1627.286691435"
Jul 09 18:30:32.505620 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:32.505583    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:40.506350 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:40.506292    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:30:40.507934 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:40.507904    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:30:41.334903 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:41.334874    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:30:41.343374 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:41.343341    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:30:50.506714 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:30:50.506668    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:31:00.506537 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:00.506490    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:31:08.828446 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:08.828411    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"]
Jul 09 18:31:10.505938 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:10.505894    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:31:20.506764 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:20.506712    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:31:23.852098 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:23.852048    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" containerID="cri-o://6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b" gracePeriod=45
Jul 09 18:31:24.335861 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.335809    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 18:31:24.604495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.604469    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:31:24.704033 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.704001    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2hkk\" (UniqueName: \"kubernetes.io/projected/a96c46fa-c37e-4ea0-883e-63bc6c561974-kube-api-access-t2hkk\") pod \"a96c46fa-c37e-4ea0-883e-63bc6c561974\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") "
Jul 09 18:31:24.704232 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.704060    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-kserve-provision-location\") pod \"a96c46fa-c37e-4ea0-883e-63bc6c561974\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") "
Jul 09 18:31:24.704232 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.704093    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-home\") pod \"a96c46fa-c37e-4ea0-883e-63bc6c561974\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") "
Jul 09 18:31:24.704232 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.704148    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-tmp-dir\") pod \"a96c46fa-c37e-4ea0-883e-63bc6c561974\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") "
Jul 09 18:31:24.704232 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.704186    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-dshm\") pod \"a96c46fa-c37e-4ea0-883e-63bc6c561974\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") "
Jul 09 18:31:24.704232 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.704217    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96c46fa-c37e-4ea0-883e-63bc6c561974-tls-certs\") pod \"a96c46fa-c37e-4ea0-883e-63bc6c561974\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") "
Jul 09 18:31:24.704586 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.704245    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-model-cache\") pod \"a96c46fa-c37e-4ea0-883e-63bc6c561974\" (UID: \"a96c46fa-c37e-4ea0-883e-63bc6c561974\") "
Jul 09 18:31:24.704799 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.704752    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-home" (OuterVolumeSpecName: "home") pod "a96c46fa-c37e-4ea0-883e-63bc6c561974" (UID: "a96c46fa-c37e-4ea0-883e-63bc6c561974"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:24.705186 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.704933    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-model-cache" (OuterVolumeSpecName: "model-cache") pod "a96c46fa-c37e-4ea0-883e-63bc6c561974" (UID: "a96c46fa-c37e-4ea0-883e-63bc6c561974"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:24.705460 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.705437    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:31:24.705569 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.705466    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:31:24.706722 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.706691    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96c46fa-c37e-4ea0-883e-63bc6c561974-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a96c46fa-c37e-4ea0-883e-63bc6c561974" (UID: "a96c46fa-c37e-4ea0-883e-63bc6c561974"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:31:24.706722 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.706707    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-dshm" (OuterVolumeSpecName: "dshm") pod "a96c46fa-c37e-4ea0-883e-63bc6c561974" (UID: "a96c46fa-c37e-4ea0-883e-63bc6c561974"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:24.706881 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.706795    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96c46fa-c37e-4ea0-883e-63bc6c561974-kube-api-access-t2hkk" (OuterVolumeSpecName: "kube-api-access-t2hkk") pod "a96c46fa-c37e-4ea0-883e-63bc6c561974" (UID: "a96c46fa-c37e-4ea0-883e-63bc6c561974"). InnerVolumeSpecName "kube-api-access-t2hkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:31:24.718089 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.718061    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a96c46fa-c37e-4ea0-883e-63bc6c561974" (UID: "a96c46fa-c37e-4ea0-883e-63bc6c561974"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:24.764089 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.764059    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a96c46fa-c37e-4ea0-883e-63bc6c561974" (UID: "a96c46fa-c37e-4ea0-883e-63bc6c561974"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:24.806121 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.806080    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:31:24.806121 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.806111    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:31:24.806121 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.806127    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96c46fa-c37e-4ea0-883e-63bc6c561974-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:31:24.806438 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.806139    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96c46fa-c37e-4ea0-883e-63bc6c561974-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:31:24.806438 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.806152    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2hkk\" (UniqueName: \"kubernetes.io/projected/a96c46fa-c37e-4ea0-883e-63bc6c561974-kube-api-access-t2hkk\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:31:24.867356 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.867255    2558 generic.go:356] "Generic (PLEG): container finished" podID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerID="6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b" exitCode=0
Jul 09 18:31:24.867356 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.867308    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"
Jul 09 18:31:24.867789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.867365    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" event={"ID":"a96c46fa-c37e-4ea0-883e-63bc6c561974","Type":"ContainerDied","Data":"6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b"}
Jul 09 18:31:24.867789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.867451    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w" event={"ID":"a96c46fa-c37e-4ea0-883e-63bc6c561974","Type":"ContainerDied","Data":"54416ef43bfc8f0e02946e6f9e1e3cd3e2343f1a5772d6294803de4e2219707a"}
Jul 09 18:31:24.867789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.867474    2558 scope.go:117] "RemoveContainer" containerID="6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b"
Jul 09 18:31:24.877314 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.877299    2558 scope.go:117] "RemoveContainer" containerID="178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a"
Jul 09 18:31:24.885200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.885181    2558 scope.go:117] "RemoveContainer" containerID="6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b"
Jul 09 18:31:24.885497 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:31:24.885476    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b\": container with ID starting with 6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b not found: ID does not exist" containerID="6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b"
Jul 09 18:31:24.885553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.885505    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b"} err="failed to get container status \"6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b\": rpc error: code = NotFound desc = could not find container \"6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b\": container with ID starting with 6d68b3fee0634064d87755163d8ccdbb0169a94a7ee629c2e875910115b9125b not found: ID does not exist"
Jul 09 18:31:24.885553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.885521    2558 scope.go:117] "RemoveContainer" containerID="178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a"
Jul 09 18:31:24.885774 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:31:24.885754    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a\": container with ID starting with 178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a not found: ID does not exist" containerID="178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a"
Jul 09 18:31:24.885818 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.885781    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a"} err="failed to get container status \"178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a\": rpc error: code = NotFound desc = could not find container \"178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a\": container with ID starting with 178fc70029c5e9ab9847c45ef0818b97e0609e3b2e06e035048c28033f88d40a not found: ID does not exist"
Jul 09 18:31:24.897114 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.897092    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"]
Jul 09 18:31:24.904701 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:24.904681    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774f4944fcx6x4w"]
Jul 09 18:31:26.893433 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:26.893403    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" path="/var/lib/kubelet/pods/a96c46fa-c37e-4ea0-883e-63bc6c561974/volumes"
Jul 09 18:31:30.506433 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:30.506392    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:31:35.580927 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.580888    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"]
Jul 09 18:31:35.581372 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.581217    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main"
Jul 09 18:31:35.581372 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.581229    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main"
Jul 09 18:31:35.581372 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.581236    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="storage-initializer"
Jul 09 18:31:35.581372 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.581242    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="storage-initializer"
Jul 09 18:31:35.581372 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.581294    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96c46fa-c37e-4ea0-883e-63bc6c561974" containerName="main"
Jul 09 18:31:35.585869 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.585849    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.588956 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.588932    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:31:35.589071 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.589049    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 09 18:31:35.597568 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.597547    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"]
Jul 09 18:31:35.698481 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.698448    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-tmp-dir\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.698481 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.698483    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-dshm\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.698729 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.698511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-home\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.698729 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.698599    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.698729 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.698640    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxcb\" (UniqueName: \"kubernetes.io/projected/37364be1-4368-4cef-90b8-ca17aa802086-kube-api-access-tpxcb\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.698729 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.698704    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-model-cache\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.698729 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.698724    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37364be1-4368-4cef-90b8-ca17aa802086-tls-certs\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.799679 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.799642    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxcb\" (UniqueName: \"kubernetes.io/projected/37364be1-4368-4cef-90b8-ca17aa802086-kube-api-access-tpxcb\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.799891 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.799703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-model-cache\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.799891 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.799724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37364be1-4368-4cef-90b8-ca17aa802086-tls-certs\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.799891 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.799750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-tmp-dir\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.799891 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.799768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-dshm\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.799891 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.799792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-home\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.799891 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.799828    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.800204 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.800184    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-tmp-dir\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.800269 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.800224    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.800269 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.800260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-model-cache\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.800423 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.800389    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-home\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.802731 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.802711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-dshm\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.802984 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.802967    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37364be1-4368-4cef-90b8-ca17aa802086-tls-certs\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.809368 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.809344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxcb\" (UniqueName: \"kubernetes.io/projected/37364be1-4368-4cef-90b8-ca17aa802086-kube-api-access-tpxcb\") pod \"custom-route-timeout-test-kserve-85b89c4fdc-dggc8\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:35.896999 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:35.896920    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:36.044892 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:36.044789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"]
Jul 09 18:31:36.047690 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:31:36.047659    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37364be1_4368_4cef_90b8_ca17aa802086.slice/crio-257c239d81408a7016363ecc9a54cb554103690448faa072fa9694e8e63d2518 WatchSource:0}: Error finding container 257c239d81408a7016363ecc9a54cb554103690448faa072fa9694e8e63d2518: Status 404 returned error can't find the container with id 257c239d81408a7016363ecc9a54cb554103690448faa072fa9694e8e63d2518
Jul 09 18:31:36.924497 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:36.924450    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" event={"ID":"37364be1-4368-4cef-90b8-ca17aa802086","Type":"ContainerStarted","Data":"b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575"}
Jul 09 18:31:36.925011 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:36.924520    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" event={"ID":"37364be1-4368-4cef-90b8-ca17aa802086","Type":"ContainerStarted","Data":"257c239d81408a7016363ecc9a54cb554103690448faa072fa9694e8e63d2518"}
Jul 09 18:31:39.941016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:39.940981    2558 generic.go:356] "Generic (PLEG): container finished" podID="37364be1-4368-4cef-90b8-ca17aa802086" containerID="b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575" exitCode=0
Jul 09 18:31:39.941480 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:39.941062    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" event={"ID":"37364be1-4368-4cef-90b8-ca17aa802086","Type":"ContainerDied","Data":"b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575"}
Jul 09 18:31:40.506844 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:40.506801    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:31:40.947309 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:40.947267    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" event={"ID":"37364be1-4368-4cef-90b8-ca17aa802086","Type":"ContainerStarted","Data":"7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0"}
Jul 09 18:31:40.982855 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:40.982797    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podStartSLOduration=2.8068728419999998 podStartE2EDuration="5.982780338s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.175907496s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:31:40.982062998 +0000 UTC m=+1696.628056547" watchObservedRunningTime="2026-07-09 18:31:40.982780338 +0000 UTC m=+1696.628773876"
Jul 09 18:31:41.897800 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:41.897761    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:45.897274 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:45.897184    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:31:45.898876 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:45.898847    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:31:50.506829 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:50.506780    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:31:55.897802 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:31:55.897763    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:32:00.506250 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:00.506193    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:32:05.897804 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:05.897748    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:32:10.506217 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:10.506164    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:32:15.897934 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:15.897893    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:32:20.506700 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:20.506656    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:32:25.898006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:25.897962    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:32:30.516465 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:30.516433    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:32:30.531570 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:30.531541    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:32:35.042780 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:35.042742    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"]
Jul 09 18:32:35.897956 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:35.897911    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:32:45.898148 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:45.898091    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:32:50.067410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.067358    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" containerID="cri-o://828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb" gracePeriod=45
Jul 09 18:32:50.517732 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.517684    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 18:32:50.810822 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.810790    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:32:50.854600 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.854562    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tls-certs\") pod \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") "
Jul 09 18:32:50.854756 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.854636    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tmp-dir\") pod \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") "
Jul 09 18:32:50.854756 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.854681    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-dshm\") pod \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") "
Jul 09 18:32:50.854831 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.854786    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") "
Jul 09 18:32:50.854876 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.854855    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-home\") pod \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") "
Jul 09 18:32:50.854916 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.854899    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-model-cache\") pod \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") "
Jul 09 18:32:50.854968 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.854934    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk2kb\" (UniqueName: \"kubernetes.io/projected/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-kube-api-access-jk2kb\") pod \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\" (UID: \"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4\") "
Jul 09 18:32:50.855163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.855122    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" (UID: "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:32:50.855712 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.855681    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-home" (OuterVolumeSpecName: "home") pod "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" (UID: "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:32:50.857162 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.857129    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-dshm" (OuterVolumeSpecName: "dshm") pod "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" (UID: "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:32:50.857781 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.857755    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-kube-api-access-jk2kb" (OuterVolumeSpecName: "kube-api-access-jk2kb") pod "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" (UID: "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4"). InnerVolumeSpecName "kube-api-access-jk2kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:32:50.857933 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.857908    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" (UID: "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:32:50.859263 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.859240    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529" (OuterVolumeSpecName: "kserve-pvc-source") pod "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" (UID: "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4"). InnerVolumeSpecName "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:32:50.869174 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.869142    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" (UID: "1e3ef116-7b8e-40a7-bf85-d97dc7225fe4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:32:50.956382 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.956322    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:32:50.956382 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.956374    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:32:50.956582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.956407    2558 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") on node \"ip-10-0-140-224.ec2.internal\" "
Jul 09 18:32:50.956582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.956420    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:32:50.956582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.956429    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:32:50.956582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.956437    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk2kb\" (UniqueName: \"kubernetes.io/projected/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-kube-api-access-jk2kb\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:32:50.956582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.956447    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:32:50.984246 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:50.984224    2558 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529") on node "ip-10-0-140-224.ec2.internal"
Jul 09 18:32:51.057349 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.057247    2558 reconciler_common.go:299] "Volume detached for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\""
Jul 09 18:32:51.277380 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.277346    2558 generic.go:356] "Generic (PLEG): container finished" podID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerID="828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb" exitCode=0
Jul 09 18:32:51.277855 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.277396    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"
Jul 09 18:32:51.277855 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.277419    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" event={"ID":"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4","Type":"ContainerDied","Data":"828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb"}
Jul 09 18:32:51.277855 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.277459    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5" event={"ID":"1e3ef116-7b8e-40a7-bf85-d97dc7225fe4","Type":"ContainerDied","Data":"7b32962cd13d5f0246d7e4d75f695f532cb85512ecef37afa4c44e2e6e0d7d23"}
Jul 09 18:32:51.277855 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.277480    2558 scope.go:117] "RemoveContainer" containerID="828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb"
Jul 09 18:32:51.285822 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.285804    2558 scope.go:117] "RemoveContainer" containerID="828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb"
Jul 09 18:32:51.286106 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:32:51.286083    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb\": container with ID starting with 828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb not found: ID does not exist" containerID="828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb"
Jul 09 18:32:51.286206 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.286113    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb"} err="failed to get container status \"828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb\": rpc error: code = NotFound desc = could not find container \"828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb\": container with ID starting with 828459f4a4a8cfcaeca7e22e26c29a5efc9068f4d503c58c72909c375ffc54cb not found: ID does not exist"
Jul 09 18:32:51.300018 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.299989    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"]
Jul 09 18:32:51.304856 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:51.304831    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dn94c5"]
Jul 09 18:32:52.894623 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:52.894590    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" path="/var/lib/kubelet/pods/1e3ef116-7b8e-40a7-bf85-d97dc7225fe4/volumes"
Jul 09 18:32:55.897787 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:55.897735    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:32:59.936538 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.936503    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"]
Jul 09 18:32:59.937025 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.936766    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main"
Jul 09 18:32:59.937025 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.936777    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main"
Jul 09 18:32:59.937025 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.936831    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3ef116-7b8e-40a7-bf85-d97dc7225fe4" containerName="main"
Jul 09 18:32:59.939749 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.939725    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:32:59.943280 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.943256    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 09 18:32:59.945984 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.945958    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-qzpk4\""
Jul 09 18:32:59.958140 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.958110    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"]
Jul 09 18:32:59.964087 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.964061    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"]
Jul 09 18:32:59.967198 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.967176    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:32:59.982819 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:32:59.982785    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"]
Jul 09 18:33:00.043524 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.043484    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.043710 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.043550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.043710 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.043582    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl5wd\" (UniqueName: \"kubernetes.io/projected/7ea07464-a811-4a56-9564-29c7ac689779-kube-api-access-kl5wd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.043710 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.043623    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea07464-a811-4a56-9564-29c7ac689779-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.050546 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.050512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a07f2df-2db1-47db-b274-9af6a54df8b1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.050757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.050716    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.050987 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.050964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7dm\" (UniqueName: \"kubernetes.io/projected/0a07f2df-2db1-47db-b274-9af6a54df8b1-kube-api-access-pq7dm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.051041 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.051024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.051093 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.051076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.051205 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.051108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.051258 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.051230    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.051300 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.051264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.151791 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.151757    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.151791 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.151791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.152045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.151814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.152045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.151835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.152045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.151969    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.152045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.152260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152048    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5wd\" (UniqueName: \"kubernetes.io/projected/7ea07464-a811-4a56-9564-29c7ac689779-kube-api-access-kl5wd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.152260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152091    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea07464-a811-4a56-9564-29c7ac689779-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.152260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a07f2df-2db1-47db-b274-9af6a54df8b1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.152260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152170    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.152260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7dm\" (UniqueName: \"kubernetes.io/projected/0a07f2df-2db1-47db-b274-9af6a54df8b1-kube-api-access-pq7dm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.152260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.152260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152241    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.152664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152296    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.152664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152368    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.152664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.152664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152585    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.152861 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.152740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.154928 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.154899    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.155064 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.154900    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.155235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.155208    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea07464-a811-4a56-9564-29c7ac689779-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.155377 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.155355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a07f2df-2db1-47db-b274-9af6a54df8b1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:00.165117 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.165096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl5wd\" (UniqueName: \"kubernetes.io/projected/7ea07464-a811-4a56-9564-29c7ac689779-kube-api-access-kl5wd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:00.165390 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:00.165367    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7dm\" (UniqueName: \"kubernetes.io/projected/0a07f2df-2db1-47db-b274-9af6a54df8b1-kube-api-access-pq7dm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:04.390449 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.390404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:04.398230 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.398195    2558 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:04.491980 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.491944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:04.492148 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.492085    2558 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:04.496482 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.496452    2558 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:04.593083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.593044    2558 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d55ac04b2c83230ef464e6ac64fdc05efc871de6bd51fec133c9f911cb36e9ef/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:04.611606 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.611573    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:04.694362 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.694292    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:04.694549 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.694441    2558 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:04.698409 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.698390    2558 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:04.719386 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.719352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:04.749392 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.749356    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:04.777297 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.777264    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:04.904720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.904687    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"]
Jul 09 18:33:04.906009 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:33:04.905980    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a07f2df_2db1_47db_b274_9af6a54df8b1.slice/crio-175057f47ad1a0b4e43c86f096abae866eb553c652856564fe4b5a85fd5bce98 WatchSource:0}: Error finding container 175057f47ad1a0b4e43c86f096abae866eb553c652856564fe4b5a85fd5bce98: Status 404 returned error can't find the container with id 175057f47ad1a0b4e43c86f096abae866eb553c652856564fe4b5a85fd5bce98
Jul 09 18:33:04.908036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.908017    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:33:04.931109 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:04.931088    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"]
Jul 09 18:33:04.933188 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:33:04.933165    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea07464_a811_4a56_9564_29c7ac689779.slice/crio-1d8bb4ecb812d0a27eb1058af16ad29fd45e8e6b122277204a53ffb6fa98dec8 WatchSource:0}: Error finding container 1d8bb4ecb812d0a27eb1058af16ad29fd45e8e6b122277204a53ffb6fa98dec8: Status 404 returned error can't find the container with id 1d8bb4ecb812d0a27eb1058af16ad29fd45e8e6b122277204a53ffb6fa98dec8
Jul 09 18:33:05.342565 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:05.342479    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" event={"ID":"0a07f2df-2db1-47db-b274-9af6a54df8b1","Type":"ContainerStarted","Data":"175057f47ad1a0b4e43c86f096abae866eb553c652856564fe4b5a85fd5bce98"}
Jul 09 18:33:05.344469 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:05.344437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" event={"ID":"7ea07464-a811-4a56-9564-29c7ac689779","Type":"ContainerStarted","Data":"f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5"}
Jul 09 18:33:05.344621 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:05.344492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" event={"ID":"7ea07464-a811-4a56-9564-29c7ac689779","Type":"ContainerStarted","Data":"1d8bb4ecb812d0a27eb1058af16ad29fd45e8e6b122277204a53ffb6fa98dec8"}
Jul 09 18:33:05.377634 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:05.377575    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podStartSLOduration=6.3775570120000005 podStartE2EDuration="6.377557012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:05.376133773 +0000 UTC m=+1781.022127313" watchObservedRunningTime="2026-07-09 18:33:05.377557012 +0000 UTC m=+1781.023550551"
Jul 09 18:33:05.897783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:05.897736    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:33:07.355915 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:07.355878    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" event={"ID":"0a07f2df-2db1-47db-b274-9af6a54df8b1","Type":"ContainerStarted","Data":"29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c"}
Jul 09 18:33:07.356415 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:07.355964    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:08.363726 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:08.363687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" event={"ID":"0a07f2df-2db1-47db-b274-9af6a54df8b1","Type":"ContainerStarted","Data":"98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081"}
Jul 09 18:33:08.390635 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:08.390577    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podStartSLOduration=7.877592255 podStartE2EDuration="9.390559549s" totalImagesPullingTime="1.512967294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:08.388513858 +0000 UTC m=+1784.034507421" watchObservedRunningTime="2026-07-09 18:33:08.390559549 +0000 UTC m=+1784.036553088"
Jul 09 18:33:14.749563 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:14.749506    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:14.750057 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:14.749656    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:14.750925 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:14.750893    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:33:14.778197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:14.778148    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:14.778408 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:14.778226    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:33:14.779763 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:14.779726    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:33:15.897792 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:15.897693    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:33:24.750646 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:24.750598    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:33:24.778378 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:24.778318    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:33:24.778756 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:24.778734    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:33:24.894980 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:24.894951    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:33:24.898183 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:24.898151    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:33:25.898275 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:25.898230    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 18:33:34.750366 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:34.750293    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:33:34.777998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:34.777955    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:33:35.908047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:35.908014    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:33:35.916197 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:35.916171    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:33:41.162926 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:41.162878    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"]
Jul 09 18:33:44.749833 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:44.749779    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:33:44.778643 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:44.778598    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:33:54.750453 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:54.750404    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:33:54.778515 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:54.778476    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:33:56.188469 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:56.188417    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" containerID="cri-o://7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0" gracePeriod=45
Jul 09 18:33:56.942255 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:56.942215    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:33:57.017201 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.017081    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-tmp-dir\") pod \"37364be1-4368-4cef-90b8-ca17aa802086\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") "
Jul 09 18:33:57.017201 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.017144    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpxcb\" (UniqueName: \"kubernetes.io/projected/37364be1-4368-4cef-90b8-ca17aa802086-kube-api-access-tpxcb\") pod \"37364be1-4368-4cef-90b8-ca17aa802086\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") "
Jul 09 18:33:57.017201 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.017184    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-model-cache\") pod \"37364be1-4368-4cef-90b8-ca17aa802086\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") "
Jul 09 18:33:57.017520 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.017234    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37364be1-4368-4cef-90b8-ca17aa802086-tls-certs\") pod \"37364be1-4368-4cef-90b8-ca17aa802086\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") "
Jul 09 18:33:57.017520 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.017274    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-kserve-provision-location\") pod \"37364be1-4368-4cef-90b8-ca17aa802086\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") "
Jul 09 18:33:57.017520 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.017344    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-dshm\") pod \"37364be1-4368-4cef-90b8-ca17aa802086\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") "
Jul 09 18:33:57.017520 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.017413    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-home\") pod \"37364be1-4368-4cef-90b8-ca17aa802086\" (UID: \"37364be1-4368-4cef-90b8-ca17aa802086\") "
Jul 09 18:33:57.017520 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.017475    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-model-cache" (OuterVolumeSpecName: "model-cache") pod "37364be1-4368-4cef-90b8-ca17aa802086" (UID: "37364be1-4368-4cef-90b8-ca17aa802086"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:57.017800 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.017700    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:33:57.018127 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.018101    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-home" (OuterVolumeSpecName: "home") pod "37364be1-4368-4cef-90b8-ca17aa802086" (UID: "37364be1-4368-4cef-90b8-ca17aa802086"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:57.019959 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.019919    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37364be1-4368-4cef-90b8-ca17aa802086-kube-api-access-tpxcb" (OuterVolumeSpecName: "kube-api-access-tpxcb") pod "37364be1-4368-4cef-90b8-ca17aa802086" (UID: "37364be1-4368-4cef-90b8-ca17aa802086"). InnerVolumeSpecName "kube-api-access-tpxcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:33:57.020522 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.020490    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-dshm" (OuterVolumeSpecName: "dshm") pod "37364be1-4368-4cef-90b8-ca17aa802086" (UID: "37364be1-4368-4cef-90b8-ca17aa802086"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:57.020642 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.020559    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37364be1-4368-4cef-90b8-ca17aa802086-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37364be1-4368-4cef-90b8-ca17aa802086" (UID: "37364be1-4368-4cef-90b8-ca17aa802086"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:33:57.037649 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.037607    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37364be1-4368-4cef-90b8-ca17aa802086" (UID: "37364be1-4368-4cef-90b8-ca17aa802086"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:57.082376 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.082313    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37364be1-4368-4cef-90b8-ca17aa802086" (UID: "37364be1-4368-4cef-90b8-ca17aa802086"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:57.118887 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.118850    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxcb\" (UniqueName: \"kubernetes.io/projected/37364be1-4368-4cef-90b8-ca17aa802086-kube-api-access-tpxcb\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:33:57.118887 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.118890    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37364be1-4368-4cef-90b8-ca17aa802086-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:33:57.119080 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.118907    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:33:57.119080 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.118921    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:33:57.119080 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.118934    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:33:57.119080 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.118948    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37364be1-4368-4cef-90b8-ca17aa802086-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:33:57.628808 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.628770    2558 generic.go:356] "Generic (PLEG): container finished" podID="37364be1-4368-4cef-90b8-ca17aa802086" containerID="7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0" exitCode=0
Jul 09 18:33:57.629209 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.628825    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"
Jul 09 18:33:57.629209 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.628859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" event={"ID":"37364be1-4368-4cef-90b8-ca17aa802086","Type":"ContainerDied","Data":"7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0"}
Jul 09 18:33:57.629209 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.628938    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" event={"ID":"37364be1-4368-4cef-90b8-ca17aa802086","Type":"ContainerDied","Data":"257c239d81408a7016363ecc9a54cb554103690448faa072fa9694e8e63d2518"}
Jul 09 18:33:57.629209 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.628971    2558 scope.go:117] "RemoveContainer" containerID="7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0"
Jul 09 18:33:57.638998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.638971    2558 scope.go:117] "RemoveContainer" containerID="b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575"
Jul 09 18:33:57.648756 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.648736    2558 scope.go:117] "RemoveContainer" containerID="7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0"
Jul 09 18:33:57.649040 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:33:57.649023    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0\": container with ID starting with 7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0 not found: ID does not exist" containerID="7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0"
Jul 09 18:33:57.649099 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.649050    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0"} err="failed to get container status \"7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0\": rpc error: code = NotFound desc = could not find container \"7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0\": container with ID starting with 7e06326e0e1efc3fac4afc68474f4fe7008f0d0517547d844a2aaa23169a44e0 not found: ID does not exist"
Jul 09 18:33:57.649099 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.649069    2558 scope.go:117] "RemoveContainer" containerID="b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575"
Jul 09 18:33:57.649351 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:33:57.649321    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575\": container with ID starting with b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575 not found: ID does not exist" containerID="b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575"
Jul 09 18:33:57.649399 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.649356    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575"} err="failed to get container status \"b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575\": rpc error: code = NotFound desc = could not find container \"b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575\": container with ID starting with b8523dd87d5b2163a75d6a1965c12c1e4f6164c8c1fd49b5e75e362aab64e575 not found: ID does not exist"
Jul 09 18:33:57.653594 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.653564    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"]
Jul 09 18:33:57.660130 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.660106    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8"]
Jul 09 18:33:57.909531 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:57.909428    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-85b89c4fdc-dggc8" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:33:58.893763 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:33:58.893721    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37364be1-4368-4cef-90b8-ca17aa802086" path="/var/lib/kubelet/pods/37364be1-4368-4cef-90b8-ca17aa802086/volumes"
Jul 09 18:34:04.750433 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:04.750367    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:34:04.778746 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:04.778687    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:34:14.750569 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:14.750520    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:34:14.778315 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:14.778277    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:34:24.750065 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:24.750017    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:34:24.778363 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:24.778311    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:34:29.563172 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.563125    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"]
Jul 09 18:34:29.563675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.563435    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="storage-initializer"
Jul 09 18:34:29.563675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.563448    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="storage-initializer"
Jul 09 18:34:29.563675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.563467    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main"
Jul 09 18:34:29.563675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.563473    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main"
Jul 09 18:34:29.563675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.563519    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="37364be1-4368-4cef-90b8-ca17aa802086" containerName="main"
Jul 09 18:34:29.566358 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.566337    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.569561 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.569541    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 09 18:34:29.569685 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.569593    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:34:29.580427 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.580403    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"]
Jul 09 18:34:29.726166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.726105    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-dshm\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.726166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.726165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-model-cache\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.726482 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.726230    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-tmp-dir\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.726482 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.726274    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzrd4\" (UniqueName: \"kubernetes.io/projected/df2000c3-2121-4ae3-819b-755fecafbcb4-kube-api-access-tzrd4\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.726482 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.726300    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-home\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.726482 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.726416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2000c3-2121-4ae3-819b-755fecafbcb4-tls-certs\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.726482 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.726460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-kserve-provision-location\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.827801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.827691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2000c3-2121-4ae3-819b-755fecafbcb4-tls-certs\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.827801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.827736    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-kserve-provision-location\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.827801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.827775    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-dshm\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.827801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.827798    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-model-cache\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.828166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.827812    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-tmp-dir\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.828166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.827829    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzrd4\" (UniqueName: \"kubernetes.io/projected/df2000c3-2121-4ae3-819b-755fecafbcb4-kube-api-access-tzrd4\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.828166 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.827853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-home\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.828359 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.828273    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-model-cache\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.828359 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.828299    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-kserve-provision-location\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.828359 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.828355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-home\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.828554 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.828400    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-tmp-dir\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.830201 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.830177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-dshm\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.830580 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.830559    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2000c3-2121-4ae3-819b-755fecafbcb4-tls-certs\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.840131 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.840103    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzrd4\" (UniqueName: \"kubernetes.io/projected/df2000c3-2121-4ae3-819b-755fecafbcb4-kube-api-access-tzrd4\") pod \"router-with-refs-test-kserve-7f59bb8c47-fvljx\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:29.876118 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:29.876066    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:30.025451 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:30.025421    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"]
Jul 09 18:34:30.028837 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:34:30.028804    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2000c3_2121_4ae3_819b_755fecafbcb4.slice/crio-eeb5a05604bf18e6748c7f604c0a96edc3291add993e2d729d305bcbf34af45a WatchSource:0}: Error finding container eeb5a05604bf18e6748c7f604c0a96edc3291add993e2d729d305bcbf34af45a: Status 404 returned error can't find the container with id eeb5a05604bf18e6748c7f604c0a96edc3291add993e2d729d305bcbf34af45a
Jul 09 18:34:30.799281 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:30.799243    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" event={"ID":"df2000c3-2121-4ae3-819b-755fecafbcb4","Type":"ContainerStarted","Data":"eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b"}
Jul 09 18:34:30.799690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:30.799309    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" event={"ID":"df2000c3-2121-4ae3-819b-755fecafbcb4","Type":"ContainerStarted","Data":"eeb5a05604bf18e6748c7f604c0a96edc3291add993e2d729d305bcbf34af45a"}
Jul 09 18:34:34.749913 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:34.749858    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:34:34.778519 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:34.778482    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:34:34.817986 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:34.817950    2558 generic.go:356] "Generic (PLEG): container finished" podID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerID="eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b" exitCode=0
Jul 09 18:34:34.818142 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:34.818023    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" event={"ID":"df2000c3-2121-4ae3-819b-755fecafbcb4","Type":"ContainerDied","Data":"eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b"}
Jul 09 18:34:35.825032 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:35.824993    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" event={"ID":"df2000c3-2121-4ae3-819b-755fecafbcb4","Type":"ContainerStarted","Data":"95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490"}
Jul 09 18:34:35.857856 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:35.857794    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podStartSLOduration=2.896559201 podStartE2EDuration="6.85777866s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.961219459s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:35.855446453 +0000 UTC m=+1871.501440012" watchObservedRunningTime="2026-07-09 18:34:35.85777866 +0000 UTC m=+1871.503772203"
Jul 09 18:34:35.876978 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:35.876945    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:39.876424 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:39.876364    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:34:39.877854 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:39.877820    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:34:44.752570 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:44.750710    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:34:44.778007 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:44.777972    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:34:49.876811 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:49.876687    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:34:54.750245 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:54.750197    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:34:54.777994 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:54.777939    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:34:59.876763 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:34:59.876706    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:35:04.750684 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:04.750631    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:35:04.777955 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:04.777906    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:35:09.877005 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:09.876953    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:35:14.750414 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:14.750354    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:35:14.777745 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:14.777700    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:35:19.877562 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:19.877516    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:35:24.749886 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:24.749839    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:35:24.778046 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:24.778001    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:35:29.877014 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:29.876966    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:35:34.749984 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:34.749935    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8001/health\": dial tcp 10.133.0.37:8001: connect: connection refused"
Jul 09 18:35:34.777671 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:34.777621    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:35:39.877432 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:39.877365    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:35:44.765778 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:44.765736    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:35:44.777798 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:44.777769    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:35:44.787466 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:44.787441    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:35:44.795774 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:44.795746    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:35:49.876732 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:49.876678    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:35:59.611202 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:59.611165    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"]
Jul 09 18:35:59.630997 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:59.630970    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"]
Jul 09 18:35:59.876846 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:35:59.876745    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:36:09.877149 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:09.877095    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:36:14.640218 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:14.640167    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" containerID="cri-o://f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5" gracePeriod=45
Jul 09 18:36:14.662498 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:14.662449    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main" containerID="cri-o://98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081" gracePeriod=45
Jul 09 18:36:14.788290 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:14.788249    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 18:36:15.775943 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.775894    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="llm-d-routing-sidecar" containerID="cri-o://29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c" gracePeriod=44
Jul 09 18:36:15.902376 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.902347    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:36:15.975271 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.975239    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-home\") pod \"7ea07464-a811-4a56-9564-29c7ac689779\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") "
Jul 09 18:36:15.975464 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.975424    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"7ea07464-a811-4a56-9564-29c7ac689779\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") "
Jul 09 18:36:15.975533 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.975482    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-dshm\") pod \"7ea07464-a811-4a56-9564-29c7ac689779\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") "
Jul 09 18:36:15.975592 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.975545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-tmp-dir\") pod \"7ea07464-a811-4a56-9564-29c7ac689779\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") "
Jul 09 18:36:15.975647 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.975594    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl5wd\" (UniqueName: \"kubernetes.io/projected/7ea07464-a811-4a56-9564-29c7ac689779-kube-api-access-kl5wd\") pod \"7ea07464-a811-4a56-9564-29c7ac689779\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") "
Jul 09 18:36:15.975647 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.975625    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea07464-a811-4a56-9564-29c7ac689779-tls-certs\") pod \"7ea07464-a811-4a56-9564-29c7ac689779\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") "
Jul 09 18:36:15.975757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.975653    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-model-cache\") pod \"7ea07464-a811-4a56-9564-29c7ac689779\" (UID: \"7ea07464-a811-4a56-9564-29c7ac689779\") "
Jul 09 18:36:15.976052 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.975933    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-home" (OuterVolumeSpecName: "home") pod "7ea07464-a811-4a56-9564-29c7ac689779" (UID: "7ea07464-a811-4a56-9564-29c7ac689779"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:15.976177 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.976096    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ea07464-a811-4a56-9564-29c7ac689779" (UID: "7ea07464-a811-4a56-9564-29c7ac689779"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:15.978443 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.978372    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea07464-a811-4a56-9564-29c7ac689779-kube-api-access-kl5wd" (OuterVolumeSpecName: "kube-api-access-kl5wd") pod "7ea07464-a811-4a56-9564-29c7ac689779" (UID: "7ea07464-a811-4a56-9564-29c7ac689779"). InnerVolumeSpecName "kube-api-access-kl5wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:36:15.978576 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.978522    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-dshm" (OuterVolumeSpecName: "dshm") pod "7ea07464-a811-4a56-9564-29c7ac689779" (UID: "7ea07464-a811-4a56-9564-29c7ac689779"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:15.978801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.978772    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea07464-a811-4a56-9564-29c7ac689779-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ea07464-a811-4a56-9564-29c7ac689779" (UID: "7ea07464-a811-4a56-9564-29c7ac689779"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:15.980261 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.980235    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529" (OuterVolumeSpecName: "kserve-pvc-source") pod "7ea07464-a811-4a56-9564-29c7ac689779" (UID: "7ea07464-a811-4a56-9564-29c7ac689779"). InnerVolumeSpecName "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:36:15.988934 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:15.988900    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ea07464-a811-4a56-9564-29c7ac689779" (UID: "7ea07464-a811-4a56-9564-29c7ac689779"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:16.021711 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.021690    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:36:16.076428 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.076396    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq7dm\" (UniqueName: \"kubernetes.io/projected/0a07f2df-2db1-47db-b274-9af6a54df8b1-kube-api-access-pq7dm\") pod \"0a07f2df-2db1-47db-b274-9af6a54df8b1\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") "
Jul 09 18:36:16.076428 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.076432    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-tmp-dir\") pod \"0a07f2df-2db1-47db-b274-9af6a54df8b1\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") "
Jul 09 18:36:16.076633 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.076451    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-dshm\") pod \"0a07f2df-2db1-47db-b274-9af6a54df8b1\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") "
Jul 09 18:36:16.076633 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.076482    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-model-cache\") pod \"0a07f2df-2db1-47db-b274-9af6a54df8b1\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") "
Jul 09 18:36:16.076633 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.076575    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"0a07f2df-2db1-47db-b274-9af6a54df8b1\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") "
Jul 09 18:36:16.076789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.076648    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-home\") pod \"0a07f2df-2db1-47db-b274-9af6a54df8b1\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") "
Jul 09 18:36:16.076789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.076699    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a07f2df-2db1-47db-b274-9af6a54df8b1-tls-certs\") pod \"0a07f2df-2db1-47db-b274-9af6a54df8b1\" (UID: \"0a07f2df-2db1-47db-b274-9af6a54df8b1\") "
Jul 09 18:36:16.076881 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.076788    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a07f2df-2db1-47db-b274-9af6a54df8b1" (UID: "0a07f2df-2db1-47db-b274-9af6a54df8b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:16.077358 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.077300    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-home" (OuterVolumeSpecName: "home") pod "0a07f2df-2db1-47db-b274-9af6a54df8b1" (UID: "0a07f2df-2db1-47db-b274-9af6a54df8b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:16.077358 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.077345    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.077358 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.077361    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.077598 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.077372    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.077598 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.077385    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.077598 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.077398    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl5wd\" (UniqueName: \"kubernetes.io/projected/7ea07464-a811-4a56-9564-29c7ac689779-kube-api-access-kl5wd\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.077598 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.077411    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea07464-a811-4a56-9564-29c7ac689779-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.077598 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.077426    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea07464-a811-4a56-9564-29c7ac689779-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.079171 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.079150    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a07f2df-2db1-47db-b274-9af6a54df8b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a07f2df-2db1-47db-b274-9af6a54df8b1" (UID: "0a07f2df-2db1-47db-b274-9af6a54df8b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:16.079285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.079247    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-dshm" (OuterVolumeSpecName: "dshm") pod "0a07f2df-2db1-47db-b274-9af6a54df8b1" (UID: "0a07f2df-2db1-47db-b274-9af6a54df8b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:16.079378 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.079316    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a07f2df-2db1-47db-b274-9af6a54df8b1-kube-api-access-pq7dm" (OuterVolumeSpecName: "kube-api-access-pq7dm") pod "0a07f2df-2db1-47db-b274-9af6a54df8b1" (UID: "0a07f2df-2db1-47db-b274-9af6a54df8b1"). InnerVolumeSpecName "kube-api-access-pq7dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:36:16.079752 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.079732    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529" (OuterVolumeSpecName: "kserve-pvc-source") pod "0a07f2df-2db1-47db-b274-9af6a54df8b1" (UID: "0a07f2df-2db1-47db-b274-9af6a54df8b1"). InnerVolumeSpecName "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:36:16.088856 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.088831    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a07f2df-2db1-47db-b274-9af6a54df8b1" (UID: "0a07f2df-2db1-47db-b274-9af6a54df8b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:16.178807 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.178771    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.178807 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.178810    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a07f2df-2db1-47db-b274-9af6a54df8b1-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.178988 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.178826    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq7dm\" (UniqueName: \"kubernetes.io/projected/0a07f2df-2db1-47db-b274-9af6a54df8b1-kube-api-access-pq7dm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.178988 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.178839    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.178988 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.178852    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a07f2df-2db1-47db-b274-9af6a54df8b1-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:36:16.178988 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.178891    2558 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") on node \"ip-10-0-140-224.ec2.internal\" "
Jul 09 18:36:16.208305 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.208286    2558 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529") on node "ip-10-0-140-224.ec2.internal"
Jul 09 18:36:16.279879 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.279843    2558 reconciler_common.go:299] "Volume detached for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\""
Jul 09 18:36:16.319986 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.319953    2558 generic.go:356] "Generic (PLEG): container finished" podID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerID="98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081" exitCode=0
Jul 09 18:36:16.319986 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.319978    2558 generic.go:356] "Generic (PLEG): container finished" podID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerID="29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c" exitCode=0
Jul 09 18:36:16.320185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.320020    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"
Jul 09 18:36:16.320185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.320046    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" event={"ID":"0a07f2df-2db1-47db-b274-9af6a54df8b1","Type":"ContainerDied","Data":"98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081"}
Jul 09 18:36:16.320185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.320148    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" event={"ID":"0a07f2df-2db1-47db-b274-9af6a54df8b1","Type":"ContainerDied","Data":"29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c"}
Jul 09 18:36:16.320185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.320177    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm" event={"ID":"0a07f2df-2db1-47db-b274-9af6a54df8b1","Type":"ContainerDied","Data":"175057f47ad1a0b4e43c86f096abae866eb553c652856564fe4b5a85fd5bce98"}
Jul 09 18:36:16.320403 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.320208    2558 scope.go:117] "RemoveContainer" containerID="98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081"
Jul 09 18:36:16.321444 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.321419    2558 generic.go:356] "Generic (PLEG): container finished" podID="7ea07464-a811-4a56-9564-29c7ac689779" containerID="f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5" exitCode=0
Jul 09 18:36:16.321582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.321461    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"
Jul 09 18:36:16.321582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.321507    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" event={"ID":"7ea07464-a811-4a56-9564-29c7ac689779","Type":"ContainerDied","Data":"f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5"}
Jul 09 18:36:16.321582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.321547    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2" event={"ID":"7ea07464-a811-4a56-9564-29c7ac689779","Type":"ContainerDied","Data":"1d8bb4ecb812d0a27eb1058af16ad29fd45e8e6b122277204a53ffb6fa98dec8"}
Jul 09 18:36:16.328903 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.328858    2558 scope.go:117] "RemoveContainer" containerID="29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c"
Jul 09 18:36:16.338489 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.338472    2558 scope.go:117] "RemoveContainer" containerID="98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081"
Jul 09 18:36:16.338780 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:36:16.338760    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081\": container with ID starting with 98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081 not found: ID does not exist" containerID="98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081"
Jul 09 18:36:16.338871 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.338786    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081"} err="failed to get container status \"98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081\": rpc error: code = NotFound desc = could not find container \"98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081\": container with ID starting with 98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081 not found: ID does not exist"
Jul 09 18:36:16.338871 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.338802    2558 scope.go:117] "RemoveContainer" containerID="29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c"
Jul 09 18:36:16.339022 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:36:16.339009    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c\": container with ID starting with 29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c not found: ID does not exist" containerID="29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c"
Jul 09 18:36:16.339063 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.339025    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c"} err="failed to get container status \"29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c\": rpc error: code = NotFound desc = could not find container \"29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c\": container with ID starting with 29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c not found: ID does not exist"
Jul 09 18:36:16.339063 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.339037    2558 scope.go:117] "RemoveContainer" containerID="98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081"
Jul 09 18:36:16.339274 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.339253    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081"} err="failed to get container status \"98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081\": rpc error: code = NotFound desc = could not find container \"98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081\": container with ID starting with 98965fdaac0254a16a68622b2a9c59b1514121a2248d35eaf59c15e0af0aa081 not found: ID does not exist"
Jul 09 18:36:16.339371 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.339276    2558 scope.go:117] "RemoveContainer" containerID="29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c"
Jul 09 18:36:16.339525 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.339507    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c"} err="failed to get container status \"29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c\": rpc error: code = NotFound desc = could not find container \"29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c\": container with ID starting with 29e3c7c7cc9f0033871613f7596c5ed317b7743bf6b092017f16c5c402a75a4c not found: ID does not exist"
Jul 09 18:36:16.339567 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.339526    2558 scope.go:117] "RemoveContainer" containerID="f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5"
Jul 09 18:36:16.349063 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.349040    2558 scope.go:117] "RemoveContainer" containerID="f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5"
Jul 09 18:36:16.349450 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:36:16.349411    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5\": container with ID starting with f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5 not found: ID does not exist" containerID="f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5"
Jul 09 18:36:16.349578 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.349458    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5"} err="failed to get container status \"f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5\": rpc error: code = NotFound desc = could not find container \"f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5\": container with ID starting with f9d37073008cfc332f610056efa1f301a50a15e0bcc54157dc108b2b686939b5 not found: ID does not exist"
Jul 09 18:36:16.349578 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.349551    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"]
Jul 09 18:36:16.354720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.354693    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wzmhm"]
Jul 09 18:36:16.368421 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.368390    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"]
Jul 09 18:36:16.375562 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.375534    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79qd8r2"]
Jul 09 18:36:16.893521 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.893483    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" path="/var/lib/kubelet/pods/0a07f2df-2db1-47db-b274-9af6a54df8b1/volumes"
Jul 09 18:36:16.894000 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:16.893931    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ea07464-a811-4a56-9564-29c7ac689779" path="/var/lib/kubelet/pods/7ea07464-a811-4a56-9564-29c7ac689779/volumes"
Jul 09 18:36:19.876836 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:19.876796    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:36:28.325924 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.325882    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:36:28.326365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.326166    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main"
Jul 09 18:36:28.326365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.326177    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main"
Jul 09 18:36:28.326365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.326193    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main"
Jul 09 18:36:28.326365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.326198    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main"
Jul 09 18:36:28.326365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.326203    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="llm-d-routing-sidecar"
Jul 09 18:36:28.326365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.326209    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="llm-d-routing-sidecar"
Jul 09 18:36:28.326365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.326255    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="llm-d-routing-sidecar"
Jul 09 18:36:28.326365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.326263    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a07f2df-2db1-47db-b274-9af6a54df8b1" containerName="main"
Jul 09 18:36:28.326365 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.326271    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea07464-a811-4a56-9564-29c7ac689779" containerName="main"
Jul 09 18:36:28.329729 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.329705    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.333015 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.332993    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-lk9gv\""
Jul 09 18:36:28.334695 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.334678    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 09 18:36:28.350654 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.350626    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:36:28.378949 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.378919    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.379102 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.378976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.379102 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.379002    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.379102 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.379082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad45789-23da-4ba7-bfe9-c00305c94892-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.379231 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.379147    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27zl6\" (UniqueName: \"kubernetes.io/projected/1ad45789-23da-4ba7-bfe9-c00305c94892-kube-api-access-27zl6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.379231 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.379173    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.390514 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.390481    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:36:28.394526 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.394505    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.415814 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.415789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:36:28.479667 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.479631    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.479830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.479674    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.479830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.479778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.479830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.479810    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.479953 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.479939    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.480000 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.479943    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c49l\" (UniqueName: \"kubernetes.io/projected/2cab947c-531b-4f22-9d74-e8be5466678a-kube-api-access-2c49l\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.480047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.480020    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.480098 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.480050    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cab947c-531b-4f22-9d74-e8be5466678a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.480098 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.480085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad45789-23da-4ba7-bfe9-c00305c94892-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.480195 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.480092    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.480195 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.480113    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27zl6\" (UniqueName: \"kubernetes.io/projected/1ad45789-23da-4ba7-bfe9-c00305c94892-kube-api-access-27zl6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.480303 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.480232    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.480303 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.480270    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.480452 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.480372    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.480598 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.480580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.482211 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.482185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.482552 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.482535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad45789-23da-4ba7-bfe9-c00305c94892-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.496508 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.496483    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27zl6\" (UniqueName: \"kubernetes.io/projected/1ad45789-23da-4ba7-bfe9-c00305c94892-kube-api-access-27zl6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:28.581028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.580935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.581028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.580988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c49l\" (UniqueName: \"kubernetes.io/projected/2cab947c-531b-4f22-9d74-e8be5466678a-kube-api-access-2c49l\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.581028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.581025    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.581302 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.581040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cab947c-531b-4f22-9d74-e8be5466678a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.581302 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.581062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.581302 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.581109    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.581494 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.581422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.581494 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.581459    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.581494 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.581472    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.583434 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.583409    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.583572 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.583557    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cab947c-531b-4f22-9d74-e8be5466678a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:28.601107 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:28.601080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c49l\" (UniqueName: \"kubernetes.io/projected/2cab947c-531b-4f22-9d74-e8be5466678a-kube-api-access-2c49l\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:29.876850 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:29.876802    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:36:37.562912 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.562874    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:37.570094 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.570065    2558 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:37.663660 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.663624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:37.663832 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.663798    2558 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:37.667517 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.667494    2558 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:37.758194 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.758166    2558 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d55ac04b2c83230ef464e6ac64fdc05efc871de6bd51fec133c9f911cb36e9ef/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:37.770071 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.770045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:37.865281 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.865208    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:37.865425 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.865364    2558 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:37.870009 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.869991    2558 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:37.881902 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.881877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:37.939498 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:37.939473    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:38.008441 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.008409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:36:38.087462 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.087433    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:36:38.089239 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:36:38.089205    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad45789_23da_4ba7_bfe9_c00305c94892.slice/crio-3ca50bbf42627c4647fe7c8b3a831be64e63e2128910d9d6592008de3f416d4c WatchSource:0}: Error finding container 3ca50bbf42627c4647fe7c8b3a831be64e63e2128910d9d6592008de3f416d4c: Status 404 returned error can't find the container with id 3ca50bbf42627c4647fe7c8b3a831be64e63e2128910d9d6592008de3f416d4c
Jul 09 18:36:38.151135 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.151108    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:36:38.152938 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:36:38.152914    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cab947c_531b_4f22_9d74_e8be5466678a.slice/crio-fd2fe74492a5355e0e6c7af37d45d4a0f09fd9c3ef6a4536bee8f73a73aeff5e WatchSource:0}: Error finding container fd2fe74492a5355e0e6c7af37d45d4a0f09fd9c3ef6a4536bee8f73a73aeff5e: Status 404 returned error can't find the container with id fd2fe74492a5355e0e6c7af37d45d4a0f09fd9c3ef6a4536bee8f73a73aeff5e
Jul 09 18:36:38.424718 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.424604    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"1ad45789-23da-4ba7-bfe9-c00305c94892","Type":"ContainerStarted","Data":"de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f"}
Jul 09 18:36:38.424718 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.424677    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"1ad45789-23da-4ba7-bfe9-c00305c94892","Type":"ContainerStarted","Data":"3ca50bbf42627c4647fe7c8b3a831be64e63e2128910d9d6592008de3f416d4c"}
Jul 09 18:36:38.426542 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.426513    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"2cab947c-531b-4f22-9d74-e8be5466678a","Type":"ContainerStarted","Data":"ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180"}
Jul 09 18:36:38.426689 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.426568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"2cab947c-531b-4f22-9d74-e8be5466678a","Type":"ContainerStarted","Data":"fd2fe74492a5355e0e6c7af37d45d4a0f09fd9c3ef6a4536bee8f73a73aeff5e"}
Jul 09 18:36:38.446502 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.446450    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=10.446437141 podStartE2EDuration="10.446437141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:36:38.445362044 +0000 UTC m=+1994.091355581" watchObservedRunningTime="2026-07-09 18:36:38.446437141 +0000 UTC m=+1994.092430679"
Jul 09 18:36:38.467260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.467203    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=10.467188681 podStartE2EDuration="10.467188681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:36:38.466578407 +0000 UTC m=+1994.112571945" watchObservedRunningTime="2026-07-09 18:36:38.467188681 +0000 UTC m=+1994.113182220"
Jul 09 18:36:38.940585 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:38.940544    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:39.886627 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:39.886594    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:36:39.895084 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:39.895045    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:36:47.940227 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:47.940189    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:36:47.941765 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:47.941737    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:36:57.941870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:36:57.941822    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:37:07.940208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:07.940163    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:37:08.462979 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:08.462943    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"]
Jul 09 18:37:17.940031 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:17.939973    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:37:23.487704 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:23.487652    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" containerID="cri-o://95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490" gracePeriod=45
Jul 09 18:37:23.887562 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:23.887448    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 09 18:37:24.245407 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.245383    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:37:24.321222 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.321180    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-tmp-dir\") pod \"df2000c3-2121-4ae3-819b-755fecafbcb4\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") "
Jul 09 18:37:24.321442 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.321244    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-model-cache\") pod \"df2000c3-2121-4ae3-819b-755fecafbcb4\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") "
Jul 09 18:37:24.321442 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.321294    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-home\") pod \"df2000c3-2121-4ae3-819b-755fecafbcb4\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") "
Jul 09 18:37:24.321442 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.321354    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-kserve-provision-location\") pod \"df2000c3-2121-4ae3-819b-755fecafbcb4\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") "
Jul 09 18:37:24.321442 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.321389    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-dshm\") pod \"df2000c3-2121-4ae3-819b-755fecafbcb4\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") "
Jul 09 18:37:24.321442 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.321428    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2000c3-2121-4ae3-819b-755fecafbcb4-tls-certs\") pod \"df2000c3-2121-4ae3-819b-755fecafbcb4\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") "
Jul 09 18:37:24.321736 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.321466    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzrd4\" (UniqueName: \"kubernetes.io/projected/df2000c3-2121-4ae3-819b-755fecafbcb4-kube-api-access-tzrd4\") pod \"df2000c3-2121-4ae3-819b-755fecafbcb4\" (UID: \"df2000c3-2121-4ae3-819b-755fecafbcb4\") "
Jul 09 18:37:24.323118 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.322774    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-home" (OuterVolumeSpecName: "home") pod "df2000c3-2121-4ae3-819b-755fecafbcb4" (UID: "df2000c3-2121-4ae3-819b-755fecafbcb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:24.323375 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.323306    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "df2000c3-2121-4ae3-819b-755fecafbcb4" (UID: "df2000c3-2121-4ae3-819b-755fecafbcb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:24.324994 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.324961    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2000c3-2121-4ae3-819b-755fecafbcb4-kube-api-access-tzrd4" (OuterVolumeSpecName: "kube-api-access-tzrd4") pod "df2000c3-2121-4ae3-819b-755fecafbcb4" (UID: "df2000c3-2121-4ae3-819b-755fecafbcb4"). InnerVolumeSpecName "kube-api-access-tzrd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:37:24.325452 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.325429    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-dshm" (OuterVolumeSpecName: "dshm") pod "df2000c3-2121-4ae3-819b-755fecafbcb4" (UID: "df2000c3-2121-4ae3-819b-755fecafbcb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:24.326432 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.326395    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2000c3-2121-4ae3-819b-755fecafbcb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df2000c3-2121-4ae3-819b-755fecafbcb4" (UID: "df2000c3-2121-4ae3-819b-755fecafbcb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:37:24.339793 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.339757    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df2000c3-2121-4ae3-819b-755fecafbcb4" (UID: "df2000c3-2121-4ae3-819b-755fecafbcb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:24.387718 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.387678    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df2000c3-2121-4ae3-819b-755fecafbcb4" (UID: "df2000c3-2121-4ae3-819b-755fecafbcb4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:37:24.422830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.422616    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.422830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.422658    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.422830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.422673    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.422830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.422689    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.422830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.422704    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2000c3-2121-4ae3-819b-755fecafbcb4-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.422830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.422719    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2000c3-2121-4ae3-819b-755fecafbcb4-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.422830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.422734    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzrd4\" (UniqueName: \"kubernetes.io/projected/df2000c3-2121-4ae3-819b-755fecafbcb4-kube-api-access-tzrd4\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.652472 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.652433    2558 generic.go:356] "Generic (PLEG): container finished" podID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerID="95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490" exitCode=0
Jul 09 18:37:24.652853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.652516    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"
Jul 09 18:37:24.652853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.652522    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" event={"ID":"df2000c3-2121-4ae3-819b-755fecafbcb4","Type":"ContainerDied","Data":"95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490"}
Jul 09 18:37:24.652853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.652633    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx" event={"ID":"df2000c3-2121-4ae3-819b-755fecafbcb4","Type":"ContainerDied","Data":"eeb5a05604bf18e6748c7f604c0a96edc3291add993e2d729d305bcbf34af45a"}
Jul 09 18:37:24.652853 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.652670    2558 scope.go:117] "RemoveContainer" containerID="95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490"
Jul 09 18:37:24.662203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.662186    2558 scope.go:117] "RemoveContainer" containerID="eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b"
Jul 09 18:37:24.672485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.672468    2558 scope.go:117] "RemoveContainer" containerID="95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490"
Jul 09 18:37:24.672764 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:37:24.672744    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490\": container with ID starting with 95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490 not found: ID does not exist" containerID="95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490"
Jul 09 18:37:24.672884 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.672774    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490"} err="failed to get container status \"95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490\": rpc error: code = NotFound desc = could not find container \"95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490\": container with ID starting with 95a31ea8a08122106f28d8af2d36a14d57e80856a40e18ed7d848f26b00e2490 not found: ID does not exist"
Jul 09 18:37:24.672884 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.672792    2558 scope.go:117] "RemoveContainer" containerID="eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b"
Jul 09 18:37:24.673065 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:37:24.673034    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b\": container with ID starting with eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b not found: ID does not exist" containerID="eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b"
Jul 09 18:37:24.673124 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.673076    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b"} err="failed to get container status \"eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b\": rpc error: code = NotFound desc = could not find container \"eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b\": container with ID starting with eea3b12ea3ce5dac7e8fee77f55ecb0bc2e0466e0fdb0100f606982474771b9b not found: ID does not exist"
Jul 09 18:37:24.676313 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.676288    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"]
Jul 09 18:37:24.680313 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.680294    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f59bb8c47-fvljx"]
Jul 09 18:37:24.894861 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:24.894826    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" path="/var/lib/kubelet/pods/df2000c3-2121-4ae3-819b-755fecafbcb4/volumes"
Jul 09 18:37:27.939994 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:27.939940    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:37:30.946880 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.946840    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"]
Jul 09 18:37:30.947288 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.947220    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main"
Jul 09 18:37:30.947288 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.947234    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main"
Jul 09 18:37:30.947288 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.947255    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="storage-initializer"
Jul 09 18:37:30.947288 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.947264    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="storage-initializer"
Jul 09 18:37:30.947472 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.947355    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="df2000c3-2121-4ae3-819b-755fecafbcb4" containerName="main"
Jul 09 18:37:30.950709 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.950681    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:30.954007 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.953962    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-xzpjh\""
Jul 09 18:37:30.954200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.954176    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:37:30.954483 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.954461    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 09 18:37:30.961675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.961645    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"]
Jul 09 18:37:30.974839 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.974813    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"]
Jul 09 18:37:30.978579 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.978555    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:30.982840 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.982805    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:30.982957 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.982861    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:30.983009 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.982976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:30.983061 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.983013    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:30.983061 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.983043    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:30.983149 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.983087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krzqk\" (UniqueName: \"kubernetes.io/projected/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kube-api-access-krzqk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:30.983149 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.983107    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:30.992678 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:30.992656    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"]
Jul 09 18:37:31.084224 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084188    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.084224 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084231    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8f705b-ad83-4300-884f-5897e8977534-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.084485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084250    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.084485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.084485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084293    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-685qv\" (UniqueName: \"kubernetes.io/projected/5c8f705b-ad83-4300-884f-5897e8977534-kube-api-access-685qv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.084485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.084485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084351    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.084485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084382    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.084485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084398    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.084485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084420    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.084753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.084753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084628    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.084753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084666    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krzqk\" (UniqueName: \"kubernetes.io/projected/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kube-api-access-krzqk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.084753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084715    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.084753 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084725    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.085004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.085004 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.084783    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.085142 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.085092    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.086998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.086969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.087229 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.087210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.095978 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.095952    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krzqk\" (UniqueName: \"kubernetes.io/projected/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kube-api-access-krzqk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.185542 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.185506    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8f705b-ad83-4300-884f-5897e8977534-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.185730 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.185550    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.185730 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.185585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-685qv\" (UniqueName: \"kubernetes.io/projected/5c8f705b-ad83-4300-884f-5897e8977534-kube-api-access-685qv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.185730 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.185618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.185904 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.185790    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.185904 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.185839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.186006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.185944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.186115 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.186092    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.186213 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.186170    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.186294 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.186275    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.186378 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.186306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.188413 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.188384    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.188526 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.188509    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8f705b-ad83-4300-884f-5897e8977534-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.197085 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.197008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-685qv\" (UniqueName: \"kubernetes.io/projected/5c8f705b-ad83-4300-884f-5897e8977534-kube-api-access-685qv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.264672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.264621    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.289656 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.289623    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:31.438273 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.438225    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"]
Jul 09 18:37:31.441596 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:37:31.441564    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c83ab86_6d91_4347_8cd4_15199ff11ae5.slice/crio-bfac3b1bb0ae9940e05fafebdf172b995ace8e737611f143dc31e6670e9ad735 WatchSource:0}: Error finding container bfac3b1bb0ae9940e05fafebdf172b995ace8e737611f143dc31e6670e9ad735: Status 404 returned error can't find the container with id bfac3b1bb0ae9940e05fafebdf172b995ace8e737611f143dc31e6670e9ad735
Jul 09 18:37:31.456235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.456212    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"]
Jul 09 18:37:31.458569 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:37:31.458541    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8f705b_ad83_4300_884f_5897e8977534.slice/crio-9ac3183404bc7d6b4a2ebcd162670764e991a145168d4f9e804f25a6fe6b86d8 WatchSource:0}: Error finding container 9ac3183404bc7d6b4a2ebcd162670764e991a145168d4f9e804f25a6fe6b86d8: Status 404 returned error can't find the container with id 9ac3183404bc7d6b4a2ebcd162670764e991a145168d4f9e804f25a6fe6b86d8
Jul 09 18:37:31.692637 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.692602    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" event={"ID":"5c8f705b-ad83-4300-884f-5897e8977534","Type":"ContainerStarted","Data":"616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04"}
Jul 09 18:37:31.692852 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.692662    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" event={"ID":"5c8f705b-ad83-4300-884f-5897e8977534","Type":"ContainerStarted","Data":"9ac3183404bc7d6b4a2ebcd162670764e991a145168d4f9e804f25a6fe6b86d8"}
Jul 09 18:37:31.694384 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.694351    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" event={"ID":"4c83ab86-6d91-4347-8cd4-15199ff11ae5","Type":"ContainerStarted","Data":"8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd"}
Jul 09 18:37:31.694474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.694416    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:31.694474 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:31.694443    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" event={"ID":"4c83ab86-6d91-4347-8cd4-15199ff11ae5","Type":"ContainerStarted","Data":"bfac3b1bb0ae9940e05fafebdf172b995ace8e737611f143dc31e6670e9ad735"}
Jul 09 18:37:32.702817 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:32.702759    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" event={"ID":"4c83ab86-6d91-4347-8cd4-15199ff11ae5","Type":"ContainerStarted","Data":"b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a"}
Jul 09 18:37:36.725941 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:36.725902    2558 generic.go:356] "Generic (PLEG): container finished" podID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerID="b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a" exitCode=0
Jul 09 18:37:36.726401 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:36.725984    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" event={"ID":"4c83ab86-6d91-4347-8cd4-15199ff11ae5","Type":"ContainerDied","Data":"b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a"}
Jul 09 18:37:36.727757 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:36.727736    2558 generic.go:356] "Generic (PLEG): container finished" podID="5c8f705b-ad83-4300-884f-5897e8977534" containerID="616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04" exitCode=0
Jul 09 18:37:36.727843 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:36.727765    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" event={"ID":"5c8f705b-ad83-4300-884f-5897e8977534","Type":"ContainerDied","Data":"616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04"}
Jul 09 18:37:37.735625 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:37.735582    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" event={"ID":"4c83ab86-6d91-4347-8cd4-15199ff11ae5","Type":"ContainerStarted","Data":"4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad"}
Jul 09 18:37:37.737810 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:37.737778    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" event={"ID":"5c8f705b-ad83-4300-884f-5897e8977534","Type":"ContainerStarted","Data":"2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552"}
Jul 09 18:37:37.763963 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:37.763898    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podStartSLOduration=3.229609268 podStartE2EDuration="7.763846852s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.534237584s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:37.761933929 +0000 UTC m=+2053.407927469" watchObservedRunningTime="2026-07-09 18:37:37.763846852 +0000 UTC m=+2053.409840393"
Jul 09 18:37:37.787957 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:37.787893    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podStartSLOduration=3.572851401 podStartE2EDuration="7.787872518s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.215021117s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:37.784458316 +0000 UTC m=+2053.430451856" watchObservedRunningTime="2026-07-09 18:37:37.787872518 +0000 UTC m=+2053.433866059"
Jul 09 18:37:37.940885 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:37.940828    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:37:41.265042 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:41.264992    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:41.265608 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:41.265094    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:41.266750 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:41.266715    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:37:41.284027 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:41.283996    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:37:41.290006 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:41.289976    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:41.290156 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:41.290045    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:37:41.291617 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:41.291588    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:37:47.940926 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:47.940820    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:37:51.265656 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:51.265611    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:37:51.290366 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:51.290297    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:37:57.940438 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:37:57.940381    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:38:01.266384 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:01.266190    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:38:01.291604 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:01.291557    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:38:07.940618 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:07.940570    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:38:11.265415 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:11.265352    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:38:11.290242 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:11.290200    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:38:17.941028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:17.940977    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:38:21.266238 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:21.266109    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:38:21.290369 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:21.290290    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:38:24.925251 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:24.925217    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:38:24.931396 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:24.931374    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:38:27.940439 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:27.940395    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:38:31.265533 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:31.265479    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:38:31.290653 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:31.290611    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:38:37.941986 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:37.941933    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:38:41.265719 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:41.265664    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:38:41.290700 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:41.290660    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:38:47.940889 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:47.940849    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:38:51.266040 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:51.265995    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:38:51.290656 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:51.290614    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:38:57.940397 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:38:57.940354    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:39:01.265429 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:01.265379    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:39:01.290611 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:01.290569    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:39:07.950401 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:07.950361    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:39:07.958185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:07.958157    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:39:11.265746 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:11.265700    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:39:11.289931 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:11.289897    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:39:21.266012 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:21.265901    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:39:21.290381 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:21.290321    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:39:31.265904 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:31.265845    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:39:31.290265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:31.290218    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:39:41.265209 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:41.265148    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:39:41.290299 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:41.290262    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:39:51.265541 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:51.265486    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:39:51.290206 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:39:51.290160    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:40:01.265925 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:01.265876    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:40:01.290536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:01.290498    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:40:11.265729 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:11.265680    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
Jul 09 18:40:11.290229 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:11.290193    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 09 18:40:21.274595 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:21.274564    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:40:21.286879 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:21.286857    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:40:21.299752 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:21.299730    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:40:21.308410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:21.308387    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:40:27.407282 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:27.407247    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"]
Jul 09 18:40:27.420968 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:27.420933    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"]
Jul 09 18:40:33.348764 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:33.348730    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:40:42.435169 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:42.435122    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main" containerID="cri-o://2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552" gracePeriod=45
Jul 09 18:40:42.446066 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:42.446033    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main" containerID="cri-o://4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad" gracePeriod=45
Jul 09 18:40:43.179230 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.179180    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="llm-d-routing-sidecar" containerID="cri-o://8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd" gracePeriod=45
Jul 09 18:40:43.327700 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.327674    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:40:43.402408 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.402313    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8f705b-ad83-4300-884f-5897e8977534-tls-certs\") pod \"5c8f705b-ad83-4300-884f-5897e8977534\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") "
Jul 09 18:40:43.402408 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.402384    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-home\") pod \"5c8f705b-ad83-4300-884f-5897e8977534\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") "
Jul 09 18:40:43.402607 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.402432    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-kserve-provision-location\") pod \"5c8f705b-ad83-4300-884f-5897e8977534\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") "
Jul 09 18:40:43.402607 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.402450    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-model-cache\") pod \"5c8f705b-ad83-4300-884f-5897e8977534\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") "
Jul 09 18:40:43.402607 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.402481    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-685qv\" (UniqueName: \"kubernetes.io/projected/5c8f705b-ad83-4300-884f-5897e8977534-kube-api-access-685qv\") pod \"5c8f705b-ad83-4300-884f-5897e8977534\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") "
Jul 09 18:40:43.402607 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.402508    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-dshm\") pod \"5c8f705b-ad83-4300-884f-5897e8977534\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") "
Jul 09 18:40:43.402607 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.402538    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-tmp-dir\") pod \"5c8f705b-ad83-4300-884f-5897e8977534\" (UID: \"5c8f705b-ad83-4300-884f-5897e8977534\") "
Jul 09 18:40:43.402869 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.402693    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c8f705b-ad83-4300-884f-5897e8977534" (UID: "5c8f705b-ad83-4300-884f-5897e8977534"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.402869 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.402831    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.403066 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.403048    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-home" (OuterVolumeSpecName: "home") pod "5c8f705b-ad83-4300-884f-5897e8977534" (UID: "5c8f705b-ad83-4300-884f-5897e8977534"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.405382 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.405353    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8f705b-ad83-4300-884f-5897e8977534-kube-api-access-685qv" (OuterVolumeSpecName: "kube-api-access-685qv") pod "5c8f705b-ad83-4300-884f-5897e8977534" (UID: "5c8f705b-ad83-4300-884f-5897e8977534"). InnerVolumeSpecName "kube-api-access-685qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:40:43.405500 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.405403    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-dshm" (OuterVolumeSpecName: "dshm") pod "5c8f705b-ad83-4300-884f-5897e8977534" (UID: "5c8f705b-ad83-4300-884f-5897e8977534"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.405500 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.405490    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8f705b-ad83-4300-884f-5897e8977534-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c8f705b-ad83-4300-884f-5897e8977534" (UID: "5c8f705b-ad83-4300-884f-5897e8977534"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:40:43.421862 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.421836    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c8f705b-ad83-4300-884f-5897e8977534" (UID: "5c8f705b-ad83-4300-884f-5897e8977534"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.429553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.429529    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:40:43.469494 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.469459    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c8f705b-ad83-4300-884f-5897e8977534" (UID: "5c8f705b-ad83-4300-884f-5897e8977534"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.503968 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.503937    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tls-certs\") pod \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") "
Jul 09 18:40:43.504126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.503993    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-dshm\") pod \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") "
Jul 09 18:40:43.504126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504028    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kserve-provision-location\") pod \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") "
Jul 09 18:40:43.504126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504049    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krzqk\" (UniqueName: \"kubernetes.io/projected/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kube-api-access-krzqk\") pod \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") "
Jul 09 18:40:43.504126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504071    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tmp-dir\") pod \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") "
Jul 09 18:40:43.504126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504090    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-home\") pod \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") "
Jul 09 18:40:43.504435 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504130    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-model-cache\") pod \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\" (UID: \"4c83ab86-6d91-4347-8cd4-15199ff11ae5\") "
Jul 09 18:40:43.504435 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504386    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8f705b-ad83-4300-884f-5897e8977534-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.504435 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504406    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.504435 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504420    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.504435 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504435    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-685qv\" (UniqueName: \"kubernetes.io/projected/5c8f705b-ad83-4300-884f-5897e8977534-kube-api-access-685qv\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.504699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504448    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.504699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504459    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c8f705b-ad83-4300-884f-5897e8977534-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.504699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.504663    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c83ab86-6d91-4347-8cd4-15199ff11ae5" (UID: "4c83ab86-6d91-4347-8cd4-15199ff11ae5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.505084 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.505058    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-home" (OuterVolumeSpecName: "home") pod "4c83ab86-6d91-4347-8cd4-15199ff11ae5" (UID: "4c83ab86-6d91-4347-8cd4-15199ff11ae5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.506411 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.506376    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kube-api-access-krzqk" (OuterVolumeSpecName: "kube-api-access-krzqk") pod "4c83ab86-6d91-4347-8cd4-15199ff11ae5" (UID: "4c83ab86-6d91-4347-8cd4-15199ff11ae5"). InnerVolumeSpecName "kube-api-access-krzqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:40:43.506517 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.506437    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-dshm" (OuterVolumeSpecName: "dshm") pod "4c83ab86-6d91-4347-8cd4-15199ff11ae5" (UID: "4c83ab86-6d91-4347-8cd4-15199ff11ae5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.506568 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.506547    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c83ab86-6d91-4347-8cd4-15199ff11ae5" (UID: "4c83ab86-6d91-4347-8cd4-15199ff11ae5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:40:43.515835 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.515810    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c83ab86-6d91-4347-8cd4-15199ff11ae5" (UID: "4c83ab86-6d91-4347-8cd4-15199ff11ae5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.560565 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.560528    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4c83ab86-6d91-4347-8cd4-15199ff11ae5" (UID: "4c83ab86-6d91-4347-8cd4-15199ff11ae5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:43.605408 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.605385    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.605408 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.605408    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.605544 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.605416    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.605544 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.605425    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.605544 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.605434    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krzqk\" (UniqueName: \"kubernetes.io/projected/4c83ab86-6d91-4347-8cd4-15199ff11ae5-kube-api-access-krzqk\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.605544 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.605442    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.605544 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.605452    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c83ab86-6d91-4347-8cd4-15199ff11ae5-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:43.659880 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.659804    2558 generic.go:356] "Generic (PLEG): container finished" podID="5c8f705b-ad83-4300-884f-5897e8977534" containerID="2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552" exitCode=0
Jul 09 18:40:43.659880 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.659864    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"
Jul 09 18:40:43.660081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.659893    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" event={"ID":"5c8f705b-ad83-4300-884f-5897e8977534","Type":"ContainerDied","Data":"2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552"}
Jul 09 18:40:43.660081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.659955    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng" event={"ID":"5c8f705b-ad83-4300-884f-5897e8977534","Type":"ContainerDied","Data":"9ac3183404bc7d6b4a2ebcd162670764e991a145168d4f9e804f25a6fe6b86d8"}
Jul 09 18:40:43.660081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.659982    2558 scope.go:117] "RemoveContainer" containerID="2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552"
Jul 09 18:40:43.662207 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.662185    2558 generic.go:356] "Generic (PLEG): container finished" podID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerID="4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad" exitCode=0
Jul 09 18:40:43.662312 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.662208    2558 generic.go:356] "Generic (PLEG): container finished" podID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerID="8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd" exitCode=0
Jul 09 18:40:43.662312 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.662236    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"
Jul 09 18:40:43.662312 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.662267    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" event={"ID":"4c83ab86-6d91-4347-8cd4-15199ff11ae5","Type":"ContainerDied","Data":"4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad"}
Jul 09 18:40:43.662462 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.662318    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" event={"ID":"4c83ab86-6d91-4347-8cd4-15199ff11ae5","Type":"ContainerDied","Data":"8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd"}
Jul 09 18:40:43.662462 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.662362    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz" event={"ID":"4c83ab86-6d91-4347-8cd4-15199ff11ae5","Type":"ContainerDied","Data":"bfac3b1bb0ae9940e05fafebdf172b995ace8e737611f143dc31e6670e9ad735"}
Jul 09 18:40:43.670058 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.670036    2558 scope.go:117] "RemoveContainer" containerID="616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04"
Jul 09 18:40:43.678560 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.678545    2558 scope.go:117] "RemoveContainer" containerID="2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552"
Jul 09 18:40:43.678827 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:40:43.678808    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552\": container with ID starting with 2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552 not found: ID does not exist" containerID="2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552"
Jul 09 18:40:43.678866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.678836    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552"} err="failed to get container status \"2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552\": rpc error: code = NotFound desc = could not find container \"2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552\": container with ID starting with 2a151661b7d4a9733aaabd7aef61119c4fff38a0700a4adf5d8ac6a7862b9552 not found: ID does not exist"
Jul 09 18:40:43.678866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.678852    2558 scope.go:117] "RemoveContainer" containerID="616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04"
Jul 09 18:40:43.679055 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:40:43.679041    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04\": container with ID starting with 616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04 not found: ID does not exist" containerID="616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04"
Jul 09 18:40:43.679100 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.679058    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04"} err="failed to get container status \"616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04\": rpc error: code = NotFound desc = could not find container \"616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04\": container with ID starting with 616bac2508ef0db58e25a24568f5f3c9b4595084eb5a741242b594e6a38e0f04 not found: ID does not exist"
Jul 09 18:40:43.679100 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.679077    2558 scope.go:117] "RemoveContainer" containerID="4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad"
Jul 09 18:40:43.685924 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.685904    2558 scope.go:117] "RemoveContainer" containerID="b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a"
Jul 09 18:40:43.692693 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.692668    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"]
Jul 09 18:40:43.693696 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.693682    2558 scope.go:117] "RemoveContainer" containerID="8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd"
Jul 09 18:40:43.698419 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.698396    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-678f7bf94fr9bqz"]
Jul 09 18:40:43.701656 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.701641    2558 scope.go:117] "RemoveContainer" containerID="4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad"
Jul 09 18:40:43.701977 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:40:43.701960    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad\": container with ID starting with 4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad not found: ID does not exist" containerID="4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad"
Jul 09 18:40:43.702024 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.701984    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad"} err="failed to get container status \"4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad\": rpc error: code = NotFound desc = could not find container \"4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad\": container with ID starting with 4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad not found: ID does not exist"
Jul 09 18:40:43.702024 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.702002    2558 scope.go:117] "RemoveContainer" containerID="b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a"
Jul 09 18:40:43.702261 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:40:43.702244    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a\": container with ID starting with b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a not found: ID does not exist" containerID="b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a"
Jul 09 18:40:43.702307 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.702266    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a"} err="failed to get container status \"b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a\": rpc error: code = NotFound desc = could not find container \"b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a\": container with ID starting with b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a not found: ID does not exist"
Jul 09 18:40:43.702307 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.702282    2558 scope.go:117] "RemoveContainer" containerID="8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd"
Jul 09 18:40:43.702573 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:40:43.702557    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd\": container with ID starting with 8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd not found: ID does not exist" containerID="8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd"
Jul 09 18:40:43.702624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.702576    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd"} err="failed to get container status \"8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd\": rpc error: code = NotFound desc = could not find container \"8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd\": container with ID starting with 8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd not found: ID does not exist"
Jul 09 18:40:43.702624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.702589    2558 scope.go:117] "RemoveContainer" containerID="4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad"
Jul 09 18:40:43.702791 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.702776    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad"} err="failed to get container status \"4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad\": rpc error: code = NotFound desc = could not find container \"4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad\": container with ID starting with 4399df7dde39ba07604203afd77aeb3be52bd38a57a60d6509a525abcae6f0ad not found: ID does not exist"
Jul 09 18:40:43.702877 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.702792    2558 scope.go:117] "RemoveContainer" containerID="b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a"
Jul 09 18:40:43.703010 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.702992    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a"} err="failed to get container status \"b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a\": rpc error: code = NotFound desc = could not find container \"b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a\": container with ID starting with b736060f260933c2c848f84ae7128f0563eff064296736bfafc88cec676f861a not found: ID does not exist"
Jul 09 18:40:43.703048 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.703012    2558 scope.go:117] "RemoveContainer" containerID="8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd"
Jul 09 18:40:43.703228 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.703212    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd"} err="failed to get container status \"8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd\": rpc error: code = NotFound desc = could not find container \"8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd\": container with ID starting with 8a581e294984a4e13a09bc55c6ae487dd479de27eb846a61f8eae0d31c27ebdd not found: ID does not exist"
Jul 09 18:40:43.719229 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.719207    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"]
Jul 09 18:40:43.724852 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:43.724830    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f2hxng"]
Jul 09 18:40:44.893304 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:44.893269    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" path="/var/lib/kubelet/pods/4c83ab86-6d91-4347-8cd4-15199ff11ae5/volumes"
Jul 09 18:40:44.893784 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:44.893768    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c8f705b-ad83-4300-884f-5897e8977534" path="/var/lib/kubelet/pods/5c8f705b-ad83-4300-884f-5897e8977534/volumes"
Jul 09 18:40:48.373112 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:48.373011    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" containerID="cri-o://de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f" gracePeriod=45
Jul 09 18:40:48.951935 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:48.951893    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 18:40:49.062925 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.062900    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:40:49.156639 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.156537    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-model-cache\") pod \"1ad45789-23da-4ba7-bfe9-c00305c94892\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") "
Jul 09 18:40:49.156812 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.156689    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"1ad45789-23da-4ba7-bfe9-c00305c94892\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") "
Jul 09 18:40:49.156812 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.156750    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27zl6\" (UniqueName: \"kubernetes.io/projected/1ad45789-23da-4ba7-bfe9-c00305c94892-kube-api-access-27zl6\") pod \"1ad45789-23da-4ba7-bfe9-c00305c94892\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") "
Jul 09 18:40:49.156812 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.156783    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-dshm\") pod \"1ad45789-23da-4ba7-bfe9-c00305c94892\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") "
Jul 09 18:40:49.156812 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.156804    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad45789-23da-4ba7-bfe9-c00305c94892-tls-certs\") pod \"1ad45789-23da-4ba7-bfe9-c00305c94892\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") "
Jul 09 18:40:49.157037 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.156826    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ad45789-23da-4ba7-bfe9-c00305c94892" (UID: "1ad45789-23da-4ba7-bfe9-c00305c94892"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:49.157037 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.156846    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-tmp-dir\") pod \"1ad45789-23da-4ba7-bfe9-c00305c94892\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") "
Jul 09 18:40:49.157037 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.156881    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-home\") pod \"1ad45789-23da-4ba7-bfe9-c00305c94892\" (UID: \"1ad45789-23da-4ba7-bfe9-c00305c94892\") "
Jul 09 18:40:49.157194 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.157119    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:49.157689 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.157623    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-home" (OuterVolumeSpecName: "home") pod "1ad45789-23da-4ba7-bfe9-c00305c94892" (UID: "1ad45789-23da-4ba7-bfe9-c00305c94892"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:49.159534 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.159478    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad45789-23da-4ba7-bfe9-c00305c94892-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ad45789-23da-4ba7-bfe9-c00305c94892" (UID: "1ad45789-23da-4ba7-bfe9-c00305c94892"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:40:49.159639 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.159550    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-dshm" (OuterVolumeSpecName: "dshm") pod "1ad45789-23da-4ba7-bfe9-c00305c94892" (UID: "1ad45789-23da-4ba7-bfe9-c00305c94892"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:49.159639 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.159587    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad45789-23da-4ba7-bfe9-c00305c94892-kube-api-access-27zl6" (OuterVolumeSpecName: "kube-api-access-27zl6") pod "1ad45789-23da-4ba7-bfe9-c00305c94892" (UID: "1ad45789-23da-4ba7-bfe9-c00305c94892"). InnerVolumeSpecName "kube-api-access-27zl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:40:49.159759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.159743    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529" (OuterVolumeSpecName: "kserve-pvc-source") pod "1ad45789-23da-4ba7-bfe9-c00305c94892" (UID: "1ad45789-23da-4ba7-bfe9-c00305c94892"). InnerVolumeSpecName "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:40:49.168883 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.168859    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ad45789-23da-4ba7-bfe9-c00305c94892" (UID: "1ad45789-23da-4ba7-bfe9-c00305c94892"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:40:49.257904 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.257863    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27zl6\" (UniqueName: \"kubernetes.io/projected/1ad45789-23da-4ba7-bfe9-c00305c94892-kube-api-access-27zl6\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:49.257904 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.257899    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:49.258116 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.257913    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ad45789-23da-4ba7-bfe9-c00305c94892-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:49.258116 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.257924    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:49.258116 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.257935    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ad45789-23da-4ba7-bfe9-c00305c94892-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:40:49.690203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.690164    2558 generic.go:356] "Generic (PLEG): container finished" podID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerID="de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f" exitCode=0
Jul 09 18:40:49.690668 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.690209    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 18:40:49.690668 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.690254    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"1ad45789-23da-4ba7-bfe9-c00305c94892","Type":"ContainerDied","Data":"de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f"}
Jul 09 18:40:49.690668 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.690309    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"1ad45789-23da-4ba7-bfe9-c00305c94892","Type":"ContainerDied","Data":"3ca50bbf42627c4647fe7c8b3a831be64e63e2128910d9d6592008de3f416d4c"}
Jul 09 18:40:49.690668 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.690350    2558 scope.go:117] "RemoveContainer" containerID="de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f"
Jul 09 18:40:49.699256 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.699235    2558 scope.go:117] "RemoveContainer" containerID="de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f"
Jul 09 18:40:49.699560 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:40:49.699542    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f\": container with ID starting with de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f not found: ID does not exist" containerID="de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f"
Jul 09 18:40:49.699619 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.699570    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f"} err="failed to get container status \"de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f\": rpc error: code = NotFound desc = could not find container \"de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f\": container with ID starting with de180f9a8b7ad6d5a0db376a027bb788add3833454ed7f7484b71b22776f2a8f not found: ID does not exist"
Jul 09 18:40:49.714567 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.714535    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:40:49.718802 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.718770    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 18:40:49.738598 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:49.738572    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:40:50.046277 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046199    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"]
Jul 09 18:40:50.046503 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046489    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="llm-d-routing-sidecar"
Jul 09 18:40:50.046553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046505    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="llm-d-routing-sidecar"
Jul 09 18:40:50.046553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046534    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="storage-initializer"
Jul 09 18:40:50.046553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046540    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="storage-initializer"
Jul 09 18:40:50.046553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046546    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main"
Jul 09 18:40:50.046553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046553    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046559    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046564    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046572    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046576    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046591    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="storage-initializer"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046596    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="storage-initializer"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046641    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="llm-d-routing-sidecar"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046649    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" containerName="main"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046654    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c83ab86-6d91-4347-8cd4-15199ff11ae5" containerName="main"
Jul 09 18:40:50.046699 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.046659    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c8f705b-ad83-4300-884f-5897e8977534" containerName="main"
Jul 09 18:40:50.051149 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.051132    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.056392 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.056366    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 09 18:40:50.056597 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.056583    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:40:50.056941 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.056924    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-swm4l\""
Jul 09 18:40:50.074647 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.074611    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"]
Jul 09 18:40:50.077967 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.077942    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"]
Jul 09 18:40:50.081247 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.081223    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.097038 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.097012    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"]
Jul 09 18:40:50.166084 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.166285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166091    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9x2h\" (UniqueName: \"kubernetes.io/projected/01b08102-8987-4dd2-81d1-ff1201fa5103-kube-api-access-g9x2h\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.166285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166119    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1850bd-9029-4cae-8cdb-9d05691f8726-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.166285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166159    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.166285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166182    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.166285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166225    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.166285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.166285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.166285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166287    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-home\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.166703 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.166703 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166425    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b08102-8987-4dd2-81d1-ff1201fa5103-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.166703 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166457    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.166703 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166494    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7pxm\" (UniqueName: \"kubernetes.io/projected/2d1850bd-9029-4cae-8cdb-9d05691f8726-kube-api-access-j7pxm\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.166703 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.166545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-dshm\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.267702 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.267664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.267702 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.267707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9x2h\" (UniqueName: \"kubernetes.io/projected/01b08102-8987-4dd2-81d1-ff1201fa5103-kube-api-access-g9x2h\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.267929 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.267731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1850bd-9029-4cae-8cdb-9d05691f8726-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.267929 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.267762    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.267929 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.267915    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.268111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.267968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.268111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268012    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.268111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268044    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.268111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268069    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-home\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.268111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.268410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268132    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.268410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268157    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.268410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b08102-8987-4dd2-81d1-ff1201fa5103-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.268410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.268410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268236    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7pxm\" (UniqueName: \"kubernetes.io/projected/2d1850bd-9029-4cae-8cdb-9d05691f8726-kube-api-access-j7pxm\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.268410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268300    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-dshm\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.268870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268420    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-home\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.268870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.268870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268468    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.268870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268668    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.269094 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.268936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.269212 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.269189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.270495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.270468    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.270495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.270488    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-dshm\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.270734 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.270715    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1850bd-9029-4cae-8cdb-9d05691f8726-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.270793 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.270716    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b08102-8987-4dd2-81d1-ff1201fa5103-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.277158 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.277127    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9x2h\" (UniqueName: \"kubernetes.io/projected/01b08102-8987-4dd2-81d1-ff1201fa5103-kube-api-access-g9x2h\") pod \"custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.277268 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.277176    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7pxm\" (UniqueName: \"kubernetes.io/projected/2d1850bd-9029-4cae-8cdb-9d05691f8726-kube-api-access-j7pxm\") pod \"custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.360603 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.360523    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.392080 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.392042    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:40:50.516748 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.516595    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"]
Jul 09 18:40:50.518939 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:40:50.518911    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1850bd_9029_4cae_8cdb_9d05691f8726.slice/crio-871fdf219f0144c8428e316d103797daa42cb25f14d2795afc731feed9e08db3 WatchSource:0}: Error finding container 871fdf219f0144c8428e316d103797daa42cb25f14d2795afc731feed9e08db3: Status 404 returned error can't find the container with id 871fdf219f0144c8428e316d103797daa42cb25f14d2795afc731feed9e08db3
Jul 09 18:40:50.541393 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.541312    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"]
Jul 09 18:40:50.549534 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:40:50.549506    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b08102_8987_4dd2_81d1_ff1201fa5103.slice/crio-b1fff9ee2735668abbe71367ead2a12b08e99722b3921c82365471cc9ace036e WatchSource:0}: Error finding container b1fff9ee2735668abbe71367ead2a12b08e99722b3921c82365471cc9ace036e: Status 404 returned error can't find the container with id b1fff9ee2735668abbe71367ead2a12b08e99722b3921c82365471cc9ace036e
Jul 09 18:40:50.697368 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.697312    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" event={"ID":"01b08102-8987-4dd2-81d1-ff1201fa5103","Type":"ContainerStarted","Data":"0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b"}
Jul 09 18:40:50.697833 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.697391    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" event={"ID":"01b08102-8987-4dd2-81d1-ff1201fa5103","Type":"ContainerStarted","Data":"b1fff9ee2735668abbe71367ead2a12b08e99722b3921c82365471cc9ace036e"}
Jul 09 18:40:50.698832 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.698799    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" event={"ID":"2d1850bd-9029-4cae-8cdb-9d05691f8726","Type":"ContainerStarted","Data":"344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840"}
Jul 09 18:40:50.698946 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.698844    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" event={"ID":"2d1850bd-9029-4cae-8cdb-9d05691f8726","Type":"ContainerStarted","Data":"871fdf219f0144c8428e316d103797daa42cb25f14d2795afc731feed9e08db3"}
Jul 09 18:40:50.698946 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.698869    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:40:50.895569 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:50.895525    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad45789-23da-4ba7-bfe9-c00305c94892" path="/var/lib/kubelet/pods/1ad45789-23da-4ba7-bfe9-c00305c94892/volumes"
Jul 09 18:40:51.706602 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:51.706554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" event={"ID":"2d1850bd-9029-4cae-8cdb-9d05691f8726","Type":"ContainerStarted","Data":"62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda"}
Jul 09 18:40:54.725224 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:54.725183    2558 generic.go:356] "Generic (PLEG): container finished" podID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerID="62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda" exitCode=0
Jul 09 18:40:54.725685 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:54.725255    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" event={"ID":"2d1850bd-9029-4cae-8cdb-9d05691f8726","Type":"ContainerDied","Data":"62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda"}
Jul 09 18:40:54.726943 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:54.726916    2558 generic.go:356] "Generic (PLEG): container finished" podID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerID="0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b" exitCode=0
Jul 09 18:40:54.727028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:54.726957    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" event={"ID":"01b08102-8987-4dd2-81d1-ff1201fa5103","Type":"ContainerDied","Data":"0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b"}
Jul 09 18:40:55.733912 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:55.733865    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" event={"ID":"01b08102-8987-4dd2-81d1-ff1201fa5103","Type":"ContainerStarted","Data":"b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6"}
Jul 09 18:40:55.736208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:55.736184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" event={"ID":"2d1850bd-9029-4cae-8cdb-9d05691f8726","Type":"ContainerStarted","Data":"10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2"}
Jul 09 18:40:55.764745 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:55.764688    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podStartSLOduration=1.9987188420000002 podStartE2EDuration="5.76467184s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.765952998s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:40:55.763415272 +0000 UTC m=+2251.409408813" watchObservedRunningTime="2026-07-09 18:40:55.76467184 +0000 UTC m=+2251.410665378"
Jul 09 18:40:55.791593 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:40:55.791526    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podStartSLOduration=2.000678086 podStartE2EDuration="5.791500462s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.790822376s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:40:55.789555336 +0000 UTC m=+2251.435548877" watchObservedRunningTime="2026-07-09 18:40:55.791500462 +0000 UTC m=+2251.437494002"
Jul 09 18:41:00.361186 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:00.361137    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:41:00.361855 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:00.361224    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:41:00.362500 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:00.362468    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:41:00.379659 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:00.379609    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:41:00.392257 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:00.392232    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:41:00.392257 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:00.392277    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:41:00.393726 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:00.393697    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:41:04.761796 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:04.761713    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="2cab947c-531b-4f22-9d74-e8be5466678a" containerName="main" containerID="cri-o://ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180" gracePeriod=45
Jul 09 18:41:05.567852 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.567820    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:41:05.711662 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.711627    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cab947c-531b-4f22-9d74-e8be5466678a-tls-certs\") pod \"2cab947c-531b-4f22-9d74-e8be5466678a\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") "
Jul 09 18:41:05.711867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.711779    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") pod \"2cab947c-531b-4f22-9d74-e8be5466678a\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") "
Jul 09 18:41:05.711867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.711832    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-dshm\") pod \"2cab947c-531b-4f22-9d74-e8be5466678a\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") "
Jul 09 18:41:05.711867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.711864    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-model-cache\") pod \"2cab947c-531b-4f22-9d74-e8be5466678a\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") "
Jul 09 18:41:05.712051 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.711927    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-home\") pod \"2cab947c-531b-4f22-9d74-e8be5466678a\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") "
Jul 09 18:41:05.712051 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.711956    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c49l\" (UniqueName: \"kubernetes.io/projected/2cab947c-531b-4f22-9d74-e8be5466678a-kube-api-access-2c49l\") pod \"2cab947c-531b-4f22-9d74-e8be5466678a\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") "
Jul 09 18:41:05.712051 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.712000    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-tmp-dir\") pod \"2cab947c-531b-4f22-9d74-e8be5466678a\" (UID: \"2cab947c-531b-4f22-9d74-e8be5466678a\") "
Jul 09 18:41:05.712650 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.712590    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cab947c-531b-4f22-9d74-e8be5466678a" (UID: "2cab947c-531b-4f22-9d74-e8be5466678a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:41:05.712860 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.712775    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-home" (OuterVolumeSpecName: "home") pod "2cab947c-531b-4f22-9d74-e8be5466678a" (UID: "2cab947c-531b-4f22-9d74-e8be5466678a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:41:05.714478 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.714444    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cab947c-531b-4f22-9d74-e8be5466678a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cab947c-531b-4f22-9d74-e8be5466678a" (UID: "2cab947c-531b-4f22-9d74-e8be5466678a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:41:05.715138 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.715103    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cab947c-531b-4f22-9d74-e8be5466678a-kube-api-access-2c49l" (OuterVolumeSpecName: "kube-api-access-2c49l") pod "2cab947c-531b-4f22-9d74-e8be5466678a" (UID: "2cab947c-531b-4f22-9d74-e8be5466678a"). InnerVolumeSpecName "kube-api-access-2c49l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:41:05.715238 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.715184    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-dshm" (OuterVolumeSpecName: "dshm") pod "2cab947c-531b-4f22-9d74-e8be5466678a" (UID: "2cab947c-531b-4f22-9d74-e8be5466678a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:41:05.715886 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.715859    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529" (OuterVolumeSpecName: "kserve-pvc-source") pod "2cab947c-531b-4f22-9d74-e8be5466678a" (UID: "2cab947c-531b-4f22-9d74-e8be5466678a"). InnerVolumeSpecName "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:41:05.724111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.724076    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cab947c-531b-4f22-9d74-e8be5466678a" (UID: "2cab947c-531b-4f22-9d74-e8be5466678a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:41:05.787860 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.787820    2558 generic.go:356] "Generic (PLEG): container finished" podID="2cab947c-531b-4f22-9d74-e8be5466678a" containerID="ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180" exitCode=0
Jul 09 18:41:05.788385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.787865    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 18:41:05.788385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.787908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"2cab947c-531b-4f22-9d74-e8be5466678a","Type":"ContainerDied","Data":"ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180"}
Jul 09 18:41:05.788385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.787976    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"2cab947c-531b-4f22-9d74-e8be5466678a","Type":"ContainerDied","Data":"fd2fe74492a5355e0e6c7af37d45d4a0f09fd9c3ef6a4536bee8f73a73aeff5e"}
Jul 09 18:41:05.788385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.788011    2558 scope.go:117] "RemoveContainer" containerID="ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180"
Jul 09 18:41:05.798606 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.798585    2558 scope.go:117] "RemoveContainer" containerID="ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180"
Jul 09 18:41:05.798913 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:41:05.798890    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180\": container with ID starting with ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180 not found: ID does not exist" containerID="ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180"
Jul 09 18:41:05.798992 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.798925    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180"} err="failed to get container status \"ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180\": rpc error: code = NotFound desc = could not find container \"ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180\": container with ID starting with ca8511e8ee64998587726ef8ca7b97ed19634ab77edbbbb6976c3d610380c180 not found: ID does not exist"
Jul 09 18:41:05.812830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.812798    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:41:05.812830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.812826    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c49l\" (UniqueName: \"kubernetes.io/projected/2cab947c-531b-4f22-9d74-e8be5466678a-kube-api-access-2c49l\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:41:05.812830 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.812837    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:41:05.813120 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.812846    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cab947c-531b-4f22-9d74-e8be5466678a-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:41:05.813120 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.812877    2558 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") on node \"ip-10-0-140-224.ec2.internal\" "
Jul 09 18:41:05.813120 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.812889    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:41:05.813120 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.812898    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cab947c-531b-4f22-9d74-e8be5466678a-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:41:05.813370 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.813349    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:41:05.817458 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.817432    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 18:41:05.845141 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.845121    2558 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529") on node "ip-10-0-140-224.ec2.internal"
Jul 09 18:41:05.913536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:05.913483    2558 reconciler_common.go:299] "Volume detached for volume \"pvc-1030d2a9-7a19-4fc9-acdf-cc609db07d6f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0459164a70e4eb529\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"csi-ee75e3a24d6771287254a6ef16fd542859f86517a956d3bc9fe2dcd8206a24f9\""
Jul 09 18:41:06.894238 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:06.894195    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cab947c-531b-4f22-9d74-e8be5466678a" path="/var/lib/kubelet/pods/2cab947c-531b-4f22-9d74-e8be5466678a/volumes"
Jul 09 18:41:10.361641 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:10.361597    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:41:10.392790 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:10.392742    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:41:12.697528 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.697490    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"]
Jul 09 18:41:12.698044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.697926    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cab947c-531b-4f22-9d74-e8be5466678a" containerName="main"
Jul 09 18:41:12.698044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.697943    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cab947c-531b-4f22-9d74-e8be5466678a" containerName="main"
Jul 09 18:41:12.698044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.698007    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cab947c-531b-4f22-9d74-e8be5466678a" containerName="main"
Jul 09 18:41:12.701354 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.701315    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.705697 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.705676    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 18:41:12.716095 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.716072    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"]
Jul 09 18:41:12.877566 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.877523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdjz\" (UniqueName: \"kubernetes.io/projected/75f8e6df-864a-4df1-b917-86151ccb2b5c-kube-api-access-7mdjz\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.877770 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.877588    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-tmp-dir\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.877770 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.877609    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-dshm\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.877770 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.877633    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f8e6df-864a-4df1-b917-86151ccb2b5c-tls-certs\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.877770 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.877655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-kserve-provision-location\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.877770 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.877673    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-home\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.877770 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.877724    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-model-cache\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.978385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-tmp-dir\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.978385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-dshm\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.978385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f8e6df-864a-4df1-b917-86151ccb2b5c-tls-certs\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.978673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-kserve-provision-location\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.978673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-home\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.978673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-model-cache\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.978673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdjz\" (UniqueName: \"kubernetes.io/projected/75f8e6df-864a-4df1-b917-86151ccb2b5c-kube-api-access-7mdjz\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.978865 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978729    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-tmp-dir\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.978922 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978889    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-kserve-provision-location\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.979009 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.978985    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-home\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.979142 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.979121    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-model-cache\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.980735 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.980711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-dshm\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.981190 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.981162    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f8e6df-864a-4df1-b917-86151ccb2b5c-tls-certs\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:12.987549 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:12.987527    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdjz\" (UniqueName: \"kubernetes.io/projected/75f8e6df-864a-4df1-b917-86151ccb2b5c-kube-api-access-7mdjz\") pod \"stop-feature-test-kserve-c58bf4d55-wfjpj\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:13.011568 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:13.011533    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:13.164788 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:13.164757    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"]
Jul 09 18:41:13.166817 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:41:13.166790    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f8e6df_864a_4df1_b917_86151ccb2b5c.slice/crio-58451707c00ae1bb37d57d4a4ee91589a050dfe7ba4291f646597546a3bceac3 WatchSource:0}: Error finding container 58451707c00ae1bb37d57d4a4ee91589a050dfe7ba4291f646597546a3bceac3: Status 404 returned error can't find the container with id 58451707c00ae1bb37d57d4a4ee91589a050dfe7ba4291f646597546a3bceac3
Jul 09 18:41:13.837958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:13.837920    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" event={"ID":"75f8e6df-864a-4df1-b917-86151ccb2b5c","Type":"ContainerStarted","Data":"0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e"}
Jul 09 18:41:13.838439 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:13.837976    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" event={"ID":"75f8e6df-864a-4df1-b917-86151ccb2b5c","Type":"ContainerStarted","Data":"58451707c00ae1bb37d57d4a4ee91589a050dfe7ba4291f646597546a3bceac3"}
Jul 09 18:41:16.856617 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:16.856583    2558 generic.go:356] "Generic (PLEG): container finished" podID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerID="0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e" exitCode=0
Jul 09 18:41:16.857057 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:16.856659    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" event={"ID":"75f8e6df-864a-4df1-b917-86151ccb2b5c","Type":"ContainerDied","Data":"0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e"}
Jul 09 18:41:17.864739 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:17.864701    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" event={"ID":"75f8e6df-864a-4df1-b917-86151ccb2b5c","Type":"ContainerStarted","Data":"f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a"}
Jul 09 18:41:17.893846 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:17.893780    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podStartSLOduration=2.3426966350000002 podStartE2EDuration="5.893762037s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.551065402s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:41:17.893368272 +0000 UTC m=+2273.539361811" watchObservedRunningTime="2026-07-09 18:41:17.893762037 +0000 UTC m=+2273.539755577"
Jul 09 18:41:18.012651 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:18.012606    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:20.362112 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:20.362062    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:41:20.392973 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:20.392937    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:41:23.012062 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:23.012021    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:41:23.013707 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:23.013674    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:41:30.361832 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:30.361764    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:41:30.393110 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:30.393067    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:41:33.012709 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:33.012653    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:41:40.361538 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:40.361490    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:41:40.392733 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:40.392677    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:41:43.013040 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:43.012995    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:41:50.361253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:50.361202    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:41:50.392660 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:50.392614    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:41:53.012625 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:41:53.012563    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:42:00.361750 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:00.361701    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:42:00.393553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:00.393503    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:42:03.013031 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:03.012984    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:42:10.361200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:10.361140    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:42:10.392999 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:10.392950    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:42:13.012715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:13.012663    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:42:20.361650 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:20.361599    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:42:20.392866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:20.392824    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:42:23.012859 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:23.012805    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:42:30.361345 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:30.361276    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:42:30.392762 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:30.392724    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:42:33.012368 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:33.012305    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:42:40.361061 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:40.361010    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:42:40.393368 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:40.393307    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:42:43.012254 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:43.012206    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:42:50.361492 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:50.361443    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:42:50.392731 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:50.392698    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:42:53.012958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:42:53.012919    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:43:00.361232 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:00.361176    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:43:00.393439 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:00.393401    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:43:03.012695 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:03.012642    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:43:10.361352 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:10.361288    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:43:10.392590 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:10.392547    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:43:13.012761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:13.012717    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:43:20.361707 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:20.361666    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:43:20.392904 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:20.392855    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:43:23.012720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:23.012672    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:43:24.953102 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:24.953071    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:43:24.959530 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:24.959509    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:43:30.361917 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:30.361856    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:43:30.392892 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:30.392855    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:43:33.012477 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:33.012436    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:43:40.361808 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:40.361757    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:43:40.392586 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:40.392547    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:43:43.012308 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:43.012262    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:43:50.361590 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:50.361474    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 09 18:43:50.393112 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:50.393077    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 09 18:43:53.012664 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:43:53.012610    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:44:00.372686 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:00.372648    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:44:00.388079 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:00.388048    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:44:00.403162 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:00.403132    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:44:00.411092 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:00.411066    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:44:03.012628 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:03.012584    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:44:13.011987 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:13.011945    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:44:16.262906 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:16.262872    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"]
Jul 09 18:44:16.268761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:16.268722    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"]
Jul 09 18:44:23.022252 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:23.022219    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:44:23.030245 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:23.030217    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:44:24.272488 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:24.272453    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"]
Jul 09 18:44:31.287645 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:31.287546    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main" containerID="cri-o://10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2" gracePeriod=45
Jul 09 18:44:31.291410 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:31.291375    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main" containerID="cri-o://b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6" gracePeriod=45
Jul 09 18:44:32.102575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.102541    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="llm-d-routing-sidecar" containerID="cri-o://344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840" gracePeriod=45
Jul 09 18:44:32.234060 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.234037    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:44:32.344491 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.344468    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:44:32.345512 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.345489    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-kserve-provision-location\") pod \"01b08102-8987-4dd2-81d1-ff1201fa5103\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") "
Jul 09 18:44:32.345621 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.345524    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b08102-8987-4dd2-81d1-ff1201fa5103-tls-certs\") pod \"01b08102-8987-4dd2-81d1-ff1201fa5103\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") "
Jul 09 18:44:32.345621 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.345581    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-dshm\") pod \"01b08102-8987-4dd2-81d1-ff1201fa5103\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") "
Jul 09 18:44:32.345718 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.345622    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-model-cache\") pod \"01b08102-8987-4dd2-81d1-ff1201fa5103\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") "
Jul 09 18:44:32.345718 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.345647    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9x2h\" (UniqueName: \"kubernetes.io/projected/01b08102-8987-4dd2-81d1-ff1201fa5103-kube-api-access-g9x2h\") pod \"01b08102-8987-4dd2-81d1-ff1201fa5103\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") "
Jul 09 18:44:32.345718 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.345663    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-home\") pod \"01b08102-8987-4dd2-81d1-ff1201fa5103\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") "
Jul 09 18:44:32.345718 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.345683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-tmp-dir\") pod \"01b08102-8987-4dd2-81d1-ff1201fa5103\" (UID: \"01b08102-8987-4dd2-81d1-ff1201fa5103\") "
Jul 09 18:44:32.346155 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.346109    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-model-cache" (OuterVolumeSpecName: "model-cache") pod "01b08102-8987-4dd2-81d1-ff1201fa5103" (UID: "01b08102-8987-4dd2-81d1-ff1201fa5103"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.346427 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.346391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-home" (OuterVolumeSpecName: "home") pod "01b08102-8987-4dd2-81d1-ff1201fa5103" (UID: "01b08102-8987-4dd2-81d1-ff1201fa5103"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.348350 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.348220    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b08102-8987-4dd2-81d1-ff1201fa5103-kube-api-access-g9x2h" (OuterVolumeSpecName: "kube-api-access-g9x2h") pod "01b08102-8987-4dd2-81d1-ff1201fa5103" (UID: "01b08102-8987-4dd2-81d1-ff1201fa5103"). InnerVolumeSpecName "kube-api-access-g9x2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:44:32.348457 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.348408    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-dshm" (OuterVolumeSpecName: "dshm") pod "01b08102-8987-4dd2-81d1-ff1201fa5103" (UID: "01b08102-8987-4dd2-81d1-ff1201fa5103"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.348457 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.348411    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b08102-8987-4dd2-81d1-ff1201fa5103-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01b08102-8987-4dd2-81d1-ff1201fa5103" (UID: "01b08102-8987-4dd2-81d1-ff1201fa5103"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:44:32.359158 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.359135    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01b08102-8987-4dd2-81d1-ff1201fa5103" (UID: "01b08102-8987-4dd2-81d1-ff1201fa5103"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.412208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.412160    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01b08102-8987-4dd2-81d1-ff1201fa5103" (UID: "01b08102-8987-4dd2-81d1-ff1201fa5103"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.447068 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447040    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-kserve-provision-location\") pod \"2d1850bd-9029-4cae-8cdb-9d05691f8726\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") "
Jul 09 18:44:32.447178 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447083    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1850bd-9029-4cae-8cdb-9d05691f8726-tls-certs\") pod \"2d1850bd-9029-4cae-8cdb-9d05691f8726\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") "
Jul 09 18:44:32.447178 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447101    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-tmp-dir\") pod \"2d1850bd-9029-4cae-8cdb-9d05691f8726\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") "
Jul 09 18:44:32.447178 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447122    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-model-cache\") pod \"2d1850bd-9029-4cae-8cdb-9d05691f8726\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") "
Jul 09 18:44:32.447178 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447162    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-dshm\") pod \"2d1850bd-9029-4cae-8cdb-9d05691f8726\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") "
Jul 09 18:44:32.447402 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447194    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7pxm\" (UniqueName: \"kubernetes.io/projected/2d1850bd-9029-4cae-8cdb-9d05691f8726-kube-api-access-j7pxm\") pod \"2d1850bd-9029-4cae-8cdb-9d05691f8726\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") "
Jul 09 18:44:32.447402 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447216    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-home\") pod \"2d1850bd-9029-4cae-8cdb-9d05691f8726\" (UID: \"2d1850bd-9029-4cae-8cdb-9d05691f8726\") "
Jul 09 18:44:32.447508 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447428    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d1850bd-9029-4cae-8cdb-9d05691f8726" (UID: "2d1850bd-9029-4cae-8cdb-9d05691f8726"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.447508 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447457    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.447508 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447476    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.447508 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447491    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9x2h\" (UniqueName: \"kubernetes.io/projected/01b08102-8987-4dd2-81d1-ff1201fa5103-kube-api-access-g9x2h\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.447508 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447506    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.447776 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447543    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.447776 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447555    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b08102-8987-4dd2-81d1-ff1201fa5103-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.447776 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.447569    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01b08102-8987-4dd2-81d1-ff1201fa5103-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.448271 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.448245    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-home" (OuterVolumeSpecName: "home") pod "2d1850bd-9029-4cae-8cdb-9d05691f8726" (UID: "2d1850bd-9029-4cae-8cdb-9d05691f8726"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.449416 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.449390    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1850bd-9029-4cae-8cdb-9d05691f8726-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d1850bd-9029-4cae-8cdb-9d05691f8726" (UID: "2d1850bd-9029-4cae-8cdb-9d05691f8726"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:44:32.449557 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.449541    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1850bd-9029-4cae-8cdb-9d05691f8726-kube-api-access-j7pxm" (OuterVolumeSpecName: "kube-api-access-j7pxm") pod "2d1850bd-9029-4cae-8cdb-9d05691f8726" (UID: "2d1850bd-9029-4cae-8cdb-9d05691f8726"). InnerVolumeSpecName "kube-api-access-j7pxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:44:32.449606 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.449574    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-dshm" (OuterVolumeSpecName: "dshm") pod "2d1850bd-9029-4cae-8cdb-9d05691f8726" (UID: "2d1850bd-9029-4cae-8cdb-9d05691f8726"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.460596 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.460574    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d1850bd-9029-4cae-8cdb-9d05691f8726" (UID: "2d1850bd-9029-4cae-8cdb-9d05691f8726"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.511914 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.511865    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d1850bd-9029-4cae-8cdb-9d05691f8726" (UID: "2d1850bd-9029-4cae-8cdb-9d05691f8726"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:32.548237 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.548180    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7pxm\" (UniqueName: \"kubernetes.io/projected/2d1850bd-9029-4cae-8cdb-9d05691f8726-kube-api-access-j7pxm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.548237 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.548206    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.548237 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.548216    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.548237 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.548225    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1850bd-9029-4cae-8cdb-9d05691f8726-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.548237 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.548233    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.548237 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.548241    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.548497 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.548248    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1850bd-9029-4cae-8cdb-9d05691f8726-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:32.832246 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.832150    2558 generic.go:356] "Generic (PLEG): container finished" podID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerID="b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6" exitCode=0
Jul 09 18:44:32.832246 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.832213    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"
Jul 09 18:44:32.832501 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.832238    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" event={"ID":"01b08102-8987-4dd2-81d1-ff1201fa5103","Type":"ContainerDied","Data":"b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6"}
Jul 09 18:44:32.832501 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.832312    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp" event={"ID":"01b08102-8987-4dd2-81d1-ff1201fa5103","Type":"ContainerDied","Data":"b1fff9ee2735668abbe71367ead2a12b08e99722b3921c82365471cc9ace036e"}
Jul 09 18:44:32.832501 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.832347    2558 scope.go:117] "RemoveContainer" containerID="b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6"
Jul 09 18:44:32.834238 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.834215    2558 generic.go:356] "Generic (PLEG): container finished" podID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerID="10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2" exitCode=0
Jul 09 18:44:32.834238 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.834235    2558 generic.go:356] "Generic (PLEG): container finished" podID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerID="344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840" exitCode=0
Jul 09 18:44:32.834459 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.834270    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" event={"ID":"2d1850bd-9029-4cae-8cdb-9d05691f8726","Type":"ContainerDied","Data":"10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2"}
Jul 09 18:44:32.834459 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.834304    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" event={"ID":"2d1850bd-9029-4cae-8cdb-9d05691f8726","Type":"ContainerDied","Data":"344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840"}
Jul 09 18:44:32.834459 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.834270    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"
Jul 09 18:44:32.834459 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.834366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4" event={"ID":"2d1850bd-9029-4cae-8cdb-9d05691f8726","Type":"ContainerDied","Data":"871fdf219f0144c8428e316d103797daa42cb25f14d2795afc731feed9e08db3"}
Jul 09 18:44:32.841924 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.841906    2558 scope.go:117] "RemoveContainer" containerID="0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b"
Jul 09 18:44:32.849752 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.849737    2558 scope.go:117] "RemoveContainer" containerID="b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6"
Jul 09 18:44:32.849998 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:44:32.849980    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6\": container with ID starting with b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6 not found: ID does not exist" containerID="b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6"
Jul 09 18:44:32.850055 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.850006    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6"} err="failed to get container status \"b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6\": rpc error: code = NotFound desc = could not find container \"b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6\": container with ID starting with b9fd1c4c67cd22cd564c0bc330aa6598ec7c60265d7034686fb9a1bba0e527b6 not found: ID does not exist"
Jul 09 18:44:32.850055 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.850025    2558 scope.go:117] "RemoveContainer" containerID="0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b"
Jul 09 18:44:32.850246 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:44:32.850231    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b\": container with ID starting with 0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b not found: ID does not exist" containerID="0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b"
Jul 09 18:44:32.850283 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.850250    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b"} err="failed to get container status \"0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b\": rpc error: code = NotFound desc = could not find container \"0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b\": container with ID starting with 0d6f63d459f70fd1d685b13d62c37645ed562f772c4c2b35078d4ff69284c88b not found: ID does not exist"
Jul 09 18:44:32.850283 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.850263    2558 scope.go:117] "RemoveContainer" containerID="10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2"
Jul 09 18:44:32.856218 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.856193    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"]
Jul 09 18:44:32.858242 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.858186    2558 scope.go:117] "RemoveContainer" containerID="62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda"
Jul 09 18:44:32.859717 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.859695    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7859b78c59-24gxp"]
Jul 09 18:44:32.865941 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.865919    2558 scope.go:117] "RemoveContainer" containerID="344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840"
Jul 09 18:44:32.872016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.871995    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"]
Jul 09 18:44:32.873406 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.873385    2558 scope.go:117] "RemoveContainer" containerID="10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2"
Jul 09 18:44:32.873652 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:44:32.873635    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2\": container with ID starting with 10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2 not found: ID does not exist" containerID="10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2"
Jul 09 18:44:32.873716 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.873660    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2"} err="failed to get container status \"10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2\": rpc error: code = NotFound desc = could not find container \"10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2\": container with ID starting with 10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2 not found: ID does not exist"
Jul 09 18:44:32.873716 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.873677    2558 scope.go:117] "RemoveContainer" containerID="62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda"
Jul 09 18:44:32.873936 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:44:32.873909    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda\": container with ID starting with 62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda not found: ID does not exist" containerID="62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda"
Jul 09 18:44:32.874451 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.873945    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda"} err="failed to get container status \"62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda\": rpc error: code = NotFound desc = could not find container \"62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda\": container with ID starting with 62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda not found: ID does not exist"
Jul 09 18:44:32.874451 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.874002    2558 scope.go:117] "RemoveContainer" containerID="344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840"
Jul 09 18:44:32.874619 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:44:32.874597    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840\": container with ID starting with 344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840 not found: ID does not exist" containerID="344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840"
Jul 09 18:44:32.874672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.874628    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840"} err="failed to get container status \"344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840\": rpc error: code = NotFound desc = could not find container \"344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840\": container with ID starting with 344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840 not found: ID does not exist"
Jul 09 18:44:32.874672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.874648    2558 scope.go:117] "RemoveContainer" containerID="10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2"
Jul 09 18:44:32.874919 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.874893    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2"} err="failed to get container status \"10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2\": rpc error: code = NotFound desc = could not find container \"10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2\": container with ID starting with 10288941c5445cacff3404e3f9b85740b69e9db13c04ff9a0f2d1f24307b32d2 not found: ID does not exist"
Jul 09 18:44:32.875003 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.874920    2558 scope.go:117] "RemoveContainer" containerID="62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda"
Jul 09 18:44:32.875142 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.875124    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda"} err="failed to get container status \"62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda\": rpc error: code = NotFound desc = could not find container \"62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda\": container with ID starting with 62909e896d896397dbbc94414da8c9a4328e46930aa26f324a63bf4ccdad5dda not found: ID does not exist"
Jul 09 18:44:32.875206 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.875143    2558 scope.go:117] "RemoveContainer" containerID="344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840"
Jul 09 18:44:32.875369 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.875349    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840"} err="failed to get container status \"344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840\": rpc error: code = NotFound desc = could not find container \"344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840\": container with ID starting with 344aca4a935370be00aadfb942719920f1abb4ddebe96958feaab741c5e3a840 not found: ID does not exist"
Jul 09 18:44:32.876044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.876026    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6ffc9757db-ks8b4"]
Jul 09 18:44:32.892990 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.892968    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" path="/var/lib/kubelet/pods/01b08102-8987-4dd2-81d1-ff1201fa5103/volumes"
Jul 09 18:44:32.893448 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:32.893433    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" path="/var/lib/kubelet/pods/2d1850bd-9029-4cae-8cdb-9d05691f8726/volumes"
Jul 09 18:44:39.816823 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:39.816780    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" containerID="cri-o://f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a" gracePeriod=45
Jul 09 18:44:40.023647 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.023594    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 18:44:40.474256 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.474227    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:44:40.511618 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.511592    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-dshm\") pod \"75f8e6df-864a-4df1-b917-86151ccb2b5c\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") "
Jul 09 18:44:40.511786 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.511637    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-home\") pod \"75f8e6df-864a-4df1-b917-86151ccb2b5c\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") "
Jul 09 18:44:40.511786 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.511671    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-kserve-provision-location\") pod \"75f8e6df-864a-4df1-b917-86151ccb2b5c\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") "
Jul 09 18:44:40.511786 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.511699    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f8e6df-864a-4df1-b917-86151ccb2b5c-tls-certs\") pod \"75f8e6df-864a-4df1-b917-86151ccb2b5c\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") "
Jul 09 18:44:40.511786 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.511739    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mdjz\" (UniqueName: \"kubernetes.io/projected/75f8e6df-864a-4df1-b917-86151ccb2b5c-kube-api-access-7mdjz\") pod \"75f8e6df-864a-4df1-b917-86151ccb2b5c\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") "
Jul 09 18:44:40.511786 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.511766    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-tmp-dir\") pod \"75f8e6df-864a-4df1-b917-86151ccb2b5c\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") "
Jul 09 18:44:40.512027 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.511812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-model-cache\") pod \"75f8e6df-864a-4df1-b917-86151ccb2b5c\" (UID: \"75f8e6df-864a-4df1-b917-86151ccb2b5c\") "
Jul 09 18:44:40.512316 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.512256    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "75f8e6df-864a-4df1-b917-86151ccb2b5c" (UID: "75f8e6df-864a-4df1-b917-86151ccb2b5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:40.512633 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.512588    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-home" (OuterVolumeSpecName: "home") pod "75f8e6df-864a-4df1-b917-86151ccb2b5c" (UID: "75f8e6df-864a-4df1-b917-86151ccb2b5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:40.514260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.514232    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-dshm" (OuterVolumeSpecName: "dshm") pod "75f8e6df-864a-4df1-b917-86151ccb2b5c" (UID: "75f8e6df-864a-4df1-b917-86151ccb2b5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:40.514260 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.514236    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8e6df-864a-4df1-b917-86151ccb2b5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75f8e6df-864a-4df1-b917-86151ccb2b5c" (UID: "75f8e6df-864a-4df1-b917-86151ccb2b5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:44:40.514433 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.514239    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f8e6df-864a-4df1-b917-86151ccb2b5c-kube-api-access-7mdjz" (OuterVolumeSpecName: "kube-api-access-7mdjz") pod "75f8e6df-864a-4df1-b917-86151ccb2b5c" (UID: "75f8e6df-864a-4df1-b917-86151ccb2b5c"). InnerVolumeSpecName "kube-api-access-7mdjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:44:40.530933 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.530896    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75f8e6df-864a-4df1-b917-86151ccb2b5c" (UID: "75f8e6df-864a-4df1-b917-86151ccb2b5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:40.584272 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.584197    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75f8e6df-864a-4df1-b917-86151ccb2b5c" (UID: "75f8e6df-864a-4df1-b917-86151ccb2b5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:44:40.612675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.612648    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:40.612675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.612672    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f8e6df-864a-4df1-b917-86151ccb2b5c-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:40.612818 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.612682    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mdjz\" (UniqueName: \"kubernetes.io/projected/75f8e6df-864a-4df1-b917-86151ccb2b5c-kube-api-access-7mdjz\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:40.612818 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.612690    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:40.612818 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.612699    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:40.612818 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.612706    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:40.612818 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.612713    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f8e6df-864a-4df1-b917-86151ccb2b5c-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:44:40.874111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.874023    2558 generic.go:356] "Generic (PLEG): container finished" podID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerID="f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a" exitCode=0
Jul 09 18:44:40.874111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.874075    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"
Jul 09 18:44:40.874111 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.874083    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" event={"ID":"75f8e6df-864a-4df1-b917-86151ccb2b5c","Type":"ContainerDied","Data":"f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a"}
Jul 09 18:44:40.874623 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.874140    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj" event={"ID":"75f8e6df-864a-4df1-b917-86151ccb2b5c","Type":"ContainerDied","Data":"58451707c00ae1bb37d57d4a4ee91589a050dfe7ba4291f646597546a3bceac3"}
Jul 09 18:44:40.874623 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.874168    2558 scope.go:117] "RemoveContainer" containerID="f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a"
Jul 09 18:44:40.884309 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.884276    2558 scope.go:117] "RemoveContainer" containerID="0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e"
Jul 09 18:44:40.893979 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.893952    2558 scope.go:117] "RemoveContainer" containerID="f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a"
Jul 09 18:44:40.896871 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:44:40.896842    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a\": container with ID starting with f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a not found: ID does not exist" containerID="f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a"
Jul 09 18:44:40.896999 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.896880    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a"} err="failed to get container status \"f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a\": rpc error: code = NotFound desc = could not find container \"f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a\": container with ID starting with f0b6f71082765df4eb8bb2a7ba76ea0f66fc46882a500d6c9c81973f4243a12a not found: ID does not exist"
Jul 09 18:44:40.896999 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.896904    2558 scope.go:117] "RemoveContainer" containerID="0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e"
Jul 09 18:44:40.897230 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:44:40.897207    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e\": container with ID starting with 0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e not found: ID does not exist" containerID="0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e"
Jul 09 18:44:40.897269 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.897240    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e"} err="failed to get container status \"0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e\": rpc error: code = NotFound desc = could not find container \"0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e\": container with ID starting with 0bbb0646148e5be41cf1f70ba21df32714fff195af6c95b2d5ac602d9f159f6e not found: ID does not exist"
Jul 09 18:44:40.901626 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.901605    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"]
Jul 09 18:44:40.905786 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:40.905765    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-wfjpj"]
Jul 09 18:44:42.892518 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:44:42.892481    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" path="/var/lib/kubelet/pods/75f8e6df-864a-4df1-b917-86151ccb2b5c/volumes"
Jul 09 18:45:04.195516 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195481    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"]
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195748    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="storage-initializer"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195759    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="storage-initializer"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195771    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195777    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195783    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195788    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195794    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="llm-d-routing-sidecar"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195799    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="llm-d-routing-sidecar"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195806    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="storage-initializer"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195812    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="storage-initializer"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195818    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195823    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195834    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="storage-initializer"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195840    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="storage-initializer"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195892    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f8e6df-864a-4df1-b917-86151ccb2b5c" containerName="main"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195903    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b08102-8987-4dd2-81d1-ff1201fa5103" containerName="main"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195908    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="llm-d-routing-sidecar"
Jul 09 18:45:04.195998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.195914    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1850bd-9029-4cae-8cdb-9d05691f8726" containerName="main"
Jul 09 18:45:04.198636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.198617    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.202014 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.201989    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:45:04.202252 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.202231    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 18:45:04.202721 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.202703    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:45:04.213048 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.213023    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"]
Jul 09 18:45:04.290085 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.290051    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-home\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.290085 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.290088    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tmp-dir\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.290360 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.290110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kserve-provision-location\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.290360 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.290128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-dshm\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.290360 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.290179    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thld8\" (UniqueName: \"kubernetes.io/projected/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kube-api-access-thld8\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.290360 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.290196    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tls-certs\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.290360 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.290216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-model-cache\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.390946 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.390908    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-home\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.390946 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.390946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tmp-dir\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.391294 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.390966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kserve-provision-location\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.391294 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.390983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-dshm\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.391294 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.391004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thld8\" (UniqueName: \"kubernetes.io/projected/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kube-api-access-thld8\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.391294 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.391025    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tls-certs\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.391294 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.391051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-model-cache\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.391613 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.391412    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tmp-dir\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.391613 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.391429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-home\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.391613 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.391511    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kserve-provision-location\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.397676 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.391997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-model-cache\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.397676 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.394642    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-dshm\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.398173 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.398147    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tls-certs\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.404394 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.404371    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thld8\" (UniqueName: \"kubernetes.io/projected/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kube-api-access-thld8\") pod \"stop-feature-test-kserve-c58bf4d55-bsj5v\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.509229 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.509122    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:04.654587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.654553    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"]
Jul 09 18:45:04.657080 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:45:04.657049    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70b6bb0_5976_4b42_8dd4_5912cfae3558.slice/crio-5857e0ff1b44b31cdb9316572a62c393263cdb1e5b103956fc4a10712ba536f5 WatchSource:0}: Error finding container 5857e0ff1b44b31cdb9316572a62c393263cdb1e5b103956fc4a10712ba536f5: Status 404 returned error can't find the container with id 5857e0ff1b44b31cdb9316572a62c393263cdb1e5b103956fc4a10712ba536f5
Jul 09 18:45:04.969932 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.969896    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" event={"ID":"e70b6bb0-5976-4b42-8dd4-5912cfae3558","Type":"ContainerStarted","Data":"336867a17046a33364f8a6e0935e5c5d26f5568dc77d4e64bff0e9c9b903d8f2"}
Jul 09 18:45:04.970103 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:04.969948    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" event={"ID":"e70b6bb0-5976-4b42-8dd4-5912cfae3558","Type":"ContainerStarted","Data":"5857e0ff1b44b31cdb9316572a62c393263cdb1e5b103956fc4a10712ba536f5"}
Jul 09 18:45:08.990650 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:08.990612    2558 generic.go:356] "Generic (PLEG): container finished" podID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerID="336867a17046a33364f8a6e0935e5c5d26f5568dc77d4e64bff0e9c9b903d8f2" exitCode=0
Jul 09 18:45:08.991038 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:08.990686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" event={"ID":"e70b6bb0-5976-4b42-8dd4-5912cfae3558","Type":"ContainerDied","Data":"336867a17046a33364f8a6e0935e5c5d26f5568dc77d4e64bff0e9c9b903d8f2"}
Jul 09 18:45:09.996589 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:09.996554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" event={"ID":"e70b6bb0-5976-4b42-8dd4-5912cfae3558","Type":"ContainerStarted","Data":"a02c8e7d8d4b96c8c7d06884218e77bdd0365bc421ade1d33e4190778e9f6c14"}
Jul 09 18:45:10.045624 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:10.045562    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podStartSLOduration=2.678503508 podStartE2EDuration="6.045541526s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.367038018s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:45:10.043447612 +0000 UTC m=+2505.689441153" watchObservedRunningTime="2026-07-09 18:45:10.045541526 +0000 UTC m=+2505.691535066"
Jul 09 18:45:10.509220 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:10.509193    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:14.509867 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:14.509822    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:45:14.511389 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:14.511361    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:45:24.509919 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:24.509817    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:45:26.732548 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.732505    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"]
Jul 09 18:45:26.735824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.735801    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.739722 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.739695    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-pb8xq\""
Jul 09 18:45:26.739911 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.739789    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 09 18:45:26.759071 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.759033    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"]
Jul 09 18:45:26.761288 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.761268    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.764724 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.764700    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"]
Jul 09 18:45:26.781491 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.781462    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"]
Jul 09 18:45:26.880870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.880833    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3990a744-136c-4cee-89ca-454c73680476-tls-certs\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.881049 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.880891    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.881049 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.880959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.881049 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.880991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-home\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.881049 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881017    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b4273c-4846-409c-ab22-a1c3a9516de2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.881235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lpg\" (UniqueName: \"kubernetes.io/projected/99b4273c-4846-409c-ab22-a1c3a9516de2-kube-api-access-b5lpg\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.881235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881105    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.881235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881127    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtx2\" (UniqueName: \"kubernetes.io/projected/3990a744-136c-4cee-89ca-454c73680476-kube-api-access-4dtx2\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.881235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.881235 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881206    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-dshm\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.881446 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881234    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.881446 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881260    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-home\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.881446 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.881446 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.881311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-model-cache\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.981958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.981923    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3990a744-136c-4cee-89ca-454c73680476-tls-certs\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.981958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.981965    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.982208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.981990    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.982208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982009    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-home\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.982208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982032    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b4273c-4846-409c-ab22-a1c3a9516de2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.982208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982075    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lpg\" (UniqueName: \"kubernetes.io/projected/99b4273c-4846-409c-ab22-a1c3a9516de2-kube-api-access-b5lpg\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.982208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982112    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.982208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtx2\" (UniqueName: \"kubernetes.io/projected/3990a744-136c-4cee-89ca-454c73680476-kube-api-access-4dtx2\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.982208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982160    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.982208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982199    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-dshm\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.982673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982225    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.982673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982253    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-home\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.982673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.982673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-model-cache\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.982673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982454    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.982673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982472    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-home\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.982673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982456    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.982673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-model-cache\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.983203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982770    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.983203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982855    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.983203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.982956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-home\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.983203 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.983003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.984789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.984755    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-dshm\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.984933 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.984806    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.985248 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.985225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3990a744-136c-4cee-89ca-454c73680476-tls-certs\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.985316 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.985264    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b4273c-4846-409c-ab22-a1c3a9516de2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:26.998370 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.998345    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtx2\" (UniqueName: \"kubernetes.io/projected/3990a744-136c-4cee-89ca-454c73680476-kube-api-access-4dtx2\") pod \"router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:26.998518 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:26.998492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lpg\" (UniqueName: \"kubernetes.io/projected/99b4273c-4846-409c-ab22-a1c3a9516de2-kube-api-access-b5lpg\") pod \"router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:27.046444 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:27.046405    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:27.071919 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:27.071872    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:27.212028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:27.211997    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"]
Jul 09 18:45:27.212897 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:45:27.212863    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3990a744_136c_4cee_89ca_454c73680476.slice/crio-c8f32f4f2b3179e9f42e6c4be7f4f59c73def0563a33ccba487fb9b226a1d2b5 WatchSource:0}: Error finding container c8f32f4f2b3179e9f42e6c4be7f4f59c73def0563a33ccba487fb9b226a1d2b5: Status 404 returned error can't find the container with id c8f32f4f2b3179e9f42e6c4be7f4f59c73def0563a33ccba487fb9b226a1d2b5
Jul 09 18:45:27.231870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:27.231844    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"]
Jul 09 18:45:27.233814 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:45:27.233750    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b4273c_4846_409c_ab22_a1c3a9516de2.slice/crio-62a0e1f2051a05dd0709c72f5b1d818407048e71ecbff084526fac93647f4e51 WatchSource:0}: Error finding container 62a0e1f2051a05dd0709c72f5b1d818407048e71ecbff084526fac93647f4e51: Status 404 returned error can't find the container with id 62a0e1f2051a05dd0709c72f5b1d818407048e71ecbff084526fac93647f4e51
Jul 09 18:45:28.091162 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:28.091111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" event={"ID":"3990a744-136c-4cee-89ca-454c73680476","Type":"ContainerStarted","Data":"9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4"}
Jul 09 18:45:28.091845 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:28.091765    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" event={"ID":"3990a744-136c-4cee-89ca-454c73680476","Type":"ContainerStarted","Data":"c8f32f4f2b3179e9f42e6c4be7f4f59c73def0563a33ccba487fb9b226a1d2b5"}
Jul 09 18:45:28.091845 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:28.091808    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:28.094045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:28.093992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" event={"ID":"99b4273c-4846-409c-ab22-a1c3a9516de2","Type":"ContainerStarted","Data":"f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3"}
Jul 09 18:45:28.094142 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:28.094048    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" event={"ID":"99b4273c-4846-409c-ab22-a1c3a9516de2","Type":"ContainerStarted","Data":"62a0e1f2051a05dd0709c72f5b1d818407048e71ecbff084526fac93647f4e51"}
Jul 09 18:45:29.113016 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:29.112969    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" event={"ID":"3990a744-136c-4cee-89ca-454c73680476","Type":"ContainerStarted","Data":"6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0"}
Jul 09 18:45:31.131886 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:31.131855    2558 generic.go:356] "Generic (PLEG): container finished" podID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerID="f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3" exitCode=0
Jul 09 18:45:31.132383 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:31.131940    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" event={"ID":"99b4273c-4846-409c-ab22-a1c3a9516de2","Type":"ContainerDied","Data":"f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3"}
Jul 09 18:45:32.137456 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:32.137423    2558 generic.go:356] "Generic (PLEG): container finished" podID="3990a744-136c-4cee-89ca-454c73680476" containerID="6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0" exitCode=0
Jul 09 18:45:32.137915 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:32.137510    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" event={"ID":"3990a744-136c-4cee-89ca-454c73680476","Type":"ContainerDied","Data":"6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0"}
Jul 09 18:45:32.139780 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:32.139761    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" event={"ID":"99b4273c-4846-409c-ab22-a1c3a9516de2","Type":"ContainerStarted","Data":"16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8"}
Jul 09 18:45:32.192998 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:32.192938    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podStartSLOduration=3.038268986 podStartE2EDuration="6.19291583s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.154646844s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:45:32.191509654 +0000 UTC m=+2527.837503194" watchObservedRunningTime="2026-07-09 18:45:32.19291583 +0000 UTC m=+2527.838909370"
Jul 09 18:45:33.148266 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:33.148233    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" event={"ID":"3990a744-136c-4cee-89ca-454c73680476","Type":"ContainerStarted","Data":"cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0"}
Jul 09 18:45:33.181802 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:33.181730    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podStartSLOduration=3.803877653 podStartE2EDuration="7.181709429s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.377831776s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:45:33.178409665 +0000 UTC m=+2528.824403232" watchObservedRunningTime="2026-07-09 18:45:33.181709429 +0000 UTC m=+2528.827702969"
Jul 09 18:45:34.510621 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:34.510570    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:45:37.047191 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:37.047147    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:37.047829 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:37.047230    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:37.048563 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:37.048499    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:45:37.066000 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:37.065972    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:45:37.072391 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:37.072355    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:37.072511 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:37.072421    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:45:37.073535 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:37.073505    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:45:44.510424 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:44.510357    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:45:47.047695 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:47.047648    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:45:47.072550 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:47.072505    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:45:54.510431 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:54.510376    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:45:57.047129 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:57.047065    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:45:57.073113 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:45:57.073072    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:46:04.510544 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:04.510489    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:46:07.047145 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:07.047095    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:46:07.072896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:07.072853    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:46:14.509843 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:14.509789    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:46:17.047311 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:17.047260    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:46:17.073057 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:17.073011    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:46:24.510574 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:24.510516    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:46:27.047366 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:27.047298    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:46:27.072847 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:27.072809    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:46:34.510285 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:34.510232    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:46:37.046938 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:37.046891    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:46:37.073320 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:37.073272    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:46:44.510081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:44.510033    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:46:47.046916 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:47.046810    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:46:47.072965 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:47.072923    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:46:54.509660 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:54.509615    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:46:57.047634 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:57.047587    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:46:57.073366 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:46:57.073307    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:47:04.510080 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:04.510033    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:47:07.047199 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:07.047150    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:47:07.072763 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:07.072720    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:47:14.509846 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:14.509795    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:47:17.047849 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:17.047799    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:47:17.073175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:17.073128    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:47:24.509845 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:24.509791    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:47:27.047438 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:27.047391    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:47:27.072377 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:27.072319    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:47:34.509704 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:34.509647    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:47:37.047253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:37.047196    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:47:37.072608 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:37.072563    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:47:44.519930 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:44.519881    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:47:44.528582 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:44.528550    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:47:45.746189 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:45.746151    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"]
Jul 09 18:47:47.047636 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:47.047546    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:47:47.072666 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:47.072632    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:47:57.047596 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:57.047542    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:47:57.072485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:47:57.072452    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:48:00.838688 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:00.838617    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" containerID="cri-o://a02c8e7d8d4b96c8c7d06884218e77bdd0365bc421ade1d33e4190778e9f6c14" gracePeriod=45
Jul 09 18:48:01.521602 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:01.521557    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 18:48:01.894491 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:01.894453    2558 generic.go:356] "Generic (PLEG): container finished" podID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerID="a02c8e7d8d4b96c8c7d06884218e77bdd0365bc421ade1d33e4190778e9f6c14" exitCode=0
Jul 09 18:48:01.894904 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:01.894525    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" event={"ID":"e70b6bb0-5976-4b42-8dd4-5912cfae3558","Type":"ContainerDied","Data":"a02c8e7d8d4b96c8c7d06884218e77bdd0365bc421ade1d33e4190778e9f6c14"}
Jul 09 18:48:02.011134 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.011107    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:48:02.054400 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.054371    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-home\") pod \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") "
Jul 09 18:48:02.054400 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.054406    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-dshm\") pod \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") "
Jul 09 18:48:02.054648 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.054442    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tls-certs\") pod \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") "
Jul 09 18:48:02.054648 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.054470    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kserve-provision-location\") pod \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") "
Jul 09 18:48:02.054648 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.054531    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thld8\" (UniqueName: \"kubernetes.io/projected/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kube-api-access-thld8\") pod \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") "
Jul 09 18:48:02.054648 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.054563    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-model-cache\") pod \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") "
Jul 09 18:48:02.055127 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.054999    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-model-cache" (OuterVolumeSpecName: "model-cache") pod "e70b6bb0-5976-4b42-8dd4-5912cfae3558" (UID: "e70b6bb0-5976-4b42-8dd4-5912cfae3558"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:02.055127 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.055112    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-home" (OuterVolumeSpecName: "home") pod "e70b6bb0-5976-4b42-8dd4-5912cfae3558" (UID: "e70b6bb0-5976-4b42-8dd4-5912cfae3558"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:02.056946 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.056921    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-dshm" (OuterVolumeSpecName: "dshm") pod "e70b6bb0-5976-4b42-8dd4-5912cfae3558" (UID: "e70b6bb0-5976-4b42-8dd4-5912cfae3558"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:02.057533 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.057451    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kube-api-access-thld8" (OuterVolumeSpecName: "kube-api-access-thld8") pod "e70b6bb0-5976-4b42-8dd4-5912cfae3558" (UID: "e70b6bb0-5976-4b42-8dd4-5912cfae3558"). InnerVolumeSpecName "kube-api-access-thld8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:48:02.057533 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.057500    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e70b6bb0-5976-4b42-8dd4-5912cfae3558" (UID: "e70b6bb0-5976-4b42-8dd4-5912cfae3558"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:48:02.123789 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.123691    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e70b6bb0-5976-4b42-8dd4-5912cfae3558" (UID: "e70b6bb0-5976-4b42-8dd4-5912cfae3558"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:02.155814 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.155786    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tmp-dir\") pod \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\" (UID: \"e70b6bb0-5976-4b42-8dd4-5912cfae3558\") "
Jul 09 18:48:02.155982 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.155968    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thld8\" (UniqueName: \"kubernetes.io/projected/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kube-api-access-thld8\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:02.156047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.155983    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:02.156047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.155992    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:02.156047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.155999    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:02.156047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.156008    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:02.156047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.156015    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:02.166220 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.166176    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e70b6bb0-5976-4b42-8dd4-5912cfae3558" (UID: "e70b6bb0-5976-4b42-8dd4-5912cfae3558"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:02.257199 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.257159    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70b6bb0-5976-4b42-8dd4-5912cfae3558-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:02.903605 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.903571    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v" event={"ID":"e70b6bb0-5976-4b42-8dd4-5912cfae3558","Type":"ContainerDied","Data":"5857e0ff1b44b31cdb9316572a62c393263cdb1e5b103956fc4a10712ba536f5"}
Jul 09 18:48:02.903931 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.903650    2558 scope.go:117] "RemoveContainer" containerID="a02c8e7d8d4b96c8c7d06884218e77bdd0365bc421ade1d33e4190778e9f6c14"
Jul 09 18:48:02.903931 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.903587    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"
Jul 09 18:48:02.912160 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.912140    2558 scope.go:117] "RemoveContainer" containerID="336867a17046a33364f8a6e0935e5c5d26f5568dc77d4e64bff0e9c9b903d8f2"
Jul 09 18:48:02.925769 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.925744    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"]
Jul 09 18:48:02.930363 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:02.930312    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-c58bf4d55-bsj5v"]
Jul 09 18:48:04.893078 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:04.893042    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" path="/var/lib/kubelet/pods/e70b6bb0-5976-4b42-8dd4-5912cfae3558/volumes"
Jul 09 18:48:07.047455 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:07.047407    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:48:07.073098 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:07.073055    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:48:14.875421 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.875377    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"]
Jul 09 18:48:14.875910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.875803    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="storage-initializer"
Jul 09 18:48:14.875910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.875824    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="storage-initializer"
Jul 09 18:48:14.875910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.875856    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main"
Jul 09 18:48:14.875910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.875865    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main"
Jul 09 18:48:14.876192 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.875971    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70b6bb0-5976-4b42-8dd4-5912cfae3558" containerName="main"
Jul 09 18:48:14.879675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.879653    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:14.885928 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.885905    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:48:14.886053 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.885950    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 09 18:48:14.900907 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.900874    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"]
Jul 09 18:48:14.966842 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.966811    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-dshm\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:14.967030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.966849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:14.967030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.966867    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tls-certs\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:14.967030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.966890    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-home\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:14.967030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.966937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9qb\" (UniqueName: \"kubernetes.io/projected/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kube-api-access-qx9qb\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:14.967030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.966982    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-cabundle-cert\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:14.967030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.967005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tmp-dir\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:14.967030 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:14.967028    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-model-cache\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.067921 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.067885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-dshm\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068133 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.067933    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068133 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.067960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tls-certs\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068133 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.067984    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-home\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068133 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.068021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9qb\" (UniqueName: \"kubernetes.io/projected/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kube-api-access-qx9qb\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068372 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.068178    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-cabundle-cert\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.068464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068485 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.068478    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-home\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068605 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.068568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tmp-dir\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068656 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.068620    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-model-cache\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.068895 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.068869    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-cabundle-cert\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.069041 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.069032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-model-cache\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.069041 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.069034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tmp-dir\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.070618 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.070595    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-dshm\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.070758 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.070741    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tls-certs\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.083671 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.083642    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9qb\" (UniqueName: \"kubernetes.io/projected/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kube-api-access-qx9qb\") pod \"lora-single-adapter-test-kserve-78f49d77c4-4brdb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.190027 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.189994    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:15.349440 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.349353    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"]
Jul 09 18:48:15.351755 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:48:15.351724    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca92f8bd_e3e1_41de_bfd3_b926a64e7afb.slice/crio-080d21af41d11fd151a9a2f8968b98ffb2b98573cfdd0eb905e3c243db290e3b WatchSource:0}: Error finding container 080d21af41d11fd151a9a2f8968b98ffb2b98573cfdd0eb905e3c243db290e3b: Status 404 returned error can't find the container with id 080d21af41d11fd151a9a2f8968b98ffb2b98573cfdd0eb905e3c243db290e3b
Jul 09 18:48:15.962186 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.962146    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" event={"ID":"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb","Type":"ContainerStarted","Data":"aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681"}
Jul 09 18:48:15.962680 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:15.962210    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" event={"ID":"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb","Type":"ContainerStarted","Data":"080d21af41d11fd151a9a2f8968b98ffb2b98573cfdd0eb905e3c243db290e3b"}
Jul 09 18:48:17.047420 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:17.047366    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 18:48:17.072682 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:17.072643    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 18:48:19.981003 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:19.980972    2558 generic.go:356] "Generic (PLEG): container finished" podID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerID="aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681" exitCode=0
Jul 09 18:48:19.981418 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:19.981051    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" event={"ID":"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb","Type":"ContainerDied","Data":"aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681"}
Jul 09 18:48:20.990943 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:20.990906    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" event={"ID":"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb","Type":"ContainerStarted","Data":"e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80"}
Jul 09 18:48:21.022467 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:21.022405    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podStartSLOduration=3.04777003 podStartE2EDuration="7.022385333s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.974615303s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:48:21.020200016 +0000 UTC m=+2696.666193559" watchObservedRunningTime="2026-07-09 18:48:21.022385333 +0000 UTC m=+2696.668378876"
Jul 09 18:48:21.190526 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:21.190486    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:24.981418 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:24.981390    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:48:24.987188 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:24.987167    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:48:25.191176 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:25.191135    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:48:25.192724 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:25.192688    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 18:48:27.056539 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:27.056505    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:48:27.072928 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:27.072896    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:48:27.082603 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:27.082576    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:48:27.091663 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:27.091641    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:48:35.190668 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:35.190618    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 18:48:36.572640 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:36.572606    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"]
Jul 09 18:48:36.587726 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:36.587698    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"]
Jul 09 18:48:45.190730 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:45.190682    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 18:48:51.598460 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:51.598412    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main" containerID="cri-o://16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8" gracePeriod=45
Jul 09 18:48:51.612578 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:51.612534    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main" containerID="cri-o://cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0" gracePeriod=45
Jul 09 18:48:52.424079 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.423957    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="llm-d-routing-sidecar" containerID="cri-o://9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4" gracePeriod=45
Jul 09 18:48:52.563096 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.563072    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:48:52.603782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.603197    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b4273c-4846-409c-ab22-a1c3a9516de2-tls-certs\") pod \"99b4273c-4846-409c-ab22-a1c3a9516de2\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") "
Jul 09 18:48:52.603782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.603275    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-home\") pod \"99b4273c-4846-409c-ab22-a1c3a9516de2\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") "
Jul 09 18:48:52.603782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.603356    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-tmp-dir\") pod \"99b4273c-4846-409c-ab22-a1c3a9516de2\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") "
Jul 09 18:48:52.603782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.603381    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-dshm\") pod \"99b4273c-4846-409c-ab22-a1c3a9516de2\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") "
Jul 09 18:48:52.603782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.603426    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5lpg\" (UniqueName: \"kubernetes.io/projected/99b4273c-4846-409c-ab22-a1c3a9516de2-kube-api-access-b5lpg\") pod \"99b4273c-4846-409c-ab22-a1c3a9516de2\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") "
Jul 09 18:48:52.603782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.603465    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-model-cache\") pod \"99b4273c-4846-409c-ab22-a1c3a9516de2\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") "
Jul 09 18:48:52.603782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.603503    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-kserve-provision-location\") pod \"99b4273c-4846-409c-ab22-a1c3a9516de2\" (UID: \"99b4273c-4846-409c-ab22-a1c3a9516de2\") "
Jul 09 18:48:52.605297 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.605269    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-model-cache" (OuterVolumeSpecName: "model-cache") pod "99b4273c-4846-409c-ab22-a1c3a9516de2" (UID: "99b4273c-4846-409c-ab22-a1c3a9516de2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.605498 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.605355    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-home" (OuterVolumeSpecName: "home") pod "99b4273c-4846-409c-ab22-a1c3a9516de2" (UID: "99b4273c-4846-409c-ab22-a1c3a9516de2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.616999 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.616887    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b4273c-4846-409c-ab22-a1c3a9516de2-kube-api-access-b5lpg" (OuterVolumeSpecName: "kube-api-access-b5lpg") pod "99b4273c-4846-409c-ab22-a1c3a9516de2" (UID: "99b4273c-4846-409c-ab22-a1c3a9516de2"). InnerVolumeSpecName "kube-api-access-b5lpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:48:52.616999 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.616975    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b4273c-4846-409c-ab22-a1c3a9516de2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99b4273c-4846-409c-ab22-a1c3a9516de2" (UID: "99b4273c-4846-409c-ab22-a1c3a9516de2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:48:52.619705 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.619678    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-dshm" (OuterVolumeSpecName: "dshm") pod "99b4273c-4846-409c-ab22-a1c3a9516de2" (UID: "99b4273c-4846-409c-ab22-a1c3a9516de2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.621769 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.621726    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99b4273c-4846-409c-ab22-a1c3a9516de2" (UID: "99b4273c-4846-409c-ab22-a1c3a9516de2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.671553 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.671511    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:48:52.676228 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.676198    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99b4273c-4846-409c-ab22-a1c3a9516de2" (UID: "99b4273c-4846-409c-ab22-a1c3a9516de2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.704403 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704357    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-home\") pod \"3990a744-136c-4cee-89ca-454c73680476\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") "
Jul 09 18:48:52.704403 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704400    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-model-cache\") pod \"3990a744-136c-4cee-89ca-454c73680476\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") "
Jul 09 18:48:52.704690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704433    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dtx2\" (UniqueName: \"kubernetes.io/projected/3990a744-136c-4cee-89ca-454c73680476-kube-api-access-4dtx2\") pod \"3990a744-136c-4cee-89ca-454c73680476\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") "
Jul 09 18:48:52.704690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704496    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-dshm\") pod \"3990a744-136c-4cee-89ca-454c73680476\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") "
Jul 09 18:48:52.704690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704531    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-kserve-provision-location\") pod \"3990a744-136c-4cee-89ca-454c73680476\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") "
Jul 09 18:48:52.704690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704592    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-tmp-dir\") pod \"3990a744-136c-4cee-89ca-454c73680476\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") "
Jul 09 18:48:52.704690 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704667    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-model-cache" (OuterVolumeSpecName: "model-cache") pod "3990a744-136c-4cee-89ca-454c73680476" (UID: "3990a744-136c-4cee-89ca-454c73680476"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.705083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704680    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3990a744-136c-4cee-89ca-454c73680476-tls-certs\") pod \"3990a744-136c-4cee-89ca-454c73680476\" (UID: \"3990a744-136c-4cee-89ca-454c73680476\") "
Jul 09 18:48:52.705083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704971    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.705083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.704996    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.705083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.705008    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.705083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.705022    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lpg\" (UniqueName: \"kubernetes.io/projected/99b4273c-4846-409c-ab22-a1c3a9516de2-kube-api-access-b5lpg\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.705083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.705037    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.705083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.705039    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-home" (OuterVolumeSpecName: "home") pod "3990a744-136c-4cee-89ca-454c73680476" (UID: "3990a744-136c-4cee-89ca-454c73680476"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.705083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.705051    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.705083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.705084    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b4273c-4846-409c-ab22-a1c3a9516de2-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.705605 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.705101    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b4273c-4846-409c-ab22-a1c3a9516de2-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.707954 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.707928    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3990a744-136c-4cee-89ca-454c73680476-kube-api-access-4dtx2" (OuterVolumeSpecName: "kube-api-access-4dtx2") pod "3990a744-136c-4cee-89ca-454c73680476" (UID: "3990a744-136c-4cee-89ca-454c73680476"). InnerVolumeSpecName "kube-api-access-4dtx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:48:52.708079 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.708000    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3990a744-136c-4cee-89ca-454c73680476-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3990a744-136c-4cee-89ca-454c73680476" (UID: "3990a744-136c-4cee-89ca-454c73680476"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:48:52.708720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.708684    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-dshm" (OuterVolumeSpecName: "dshm") pod "3990a744-136c-4cee-89ca-454c73680476" (UID: "3990a744-136c-4cee-89ca-454c73680476"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.720869 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.720832    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3990a744-136c-4cee-89ca-454c73680476" (UID: "3990a744-136c-4cee-89ca-454c73680476"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.769565 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.769505    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3990a744-136c-4cee-89ca-454c73680476" (UID: "3990a744-136c-4cee-89ca-454c73680476"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:48:52.806194 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.806141    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.806194 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.806181    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3990a744-136c-4cee-89ca-454c73680476-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.806194 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.806194    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.806552 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.806209    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtx2\" (UniqueName: \"kubernetes.io/projected/3990a744-136c-4cee-89ca-454c73680476-kube-api-access-4dtx2\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.806552 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.806223    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:52.806552 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:52.806236    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3990a744-136c-4cee-89ca-454c73680476-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:48:53.163239 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.163135    2558 generic.go:356] "Generic (PLEG): container finished" podID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerID="16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8" exitCode=0
Jul 09 18:48:53.163239 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.163200    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"
Jul 09 18:48:53.163239 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.163208    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" event={"ID":"99b4273c-4846-409c-ab22-a1c3a9516de2","Type":"ContainerDied","Data":"16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8"}
Jul 09 18:48:53.163576 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.163268    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s" event={"ID":"99b4273c-4846-409c-ab22-a1c3a9516de2","Type":"ContainerDied","Data":"62a0e1f2051a05dd0709c72f5b1d818407048e71ecbff084526fac93647f4e51"}
Jul 09 18:48:53.163576 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.163305    2558 scope.go:117] "RemoveContainer" containerID="16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8"
Jul 09 18:48:53.165495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.165315    2558 generic.go:356] "Generic (PLEG): container finished" podID="3990a744-136c-4cee-89ca-454c73680476" containerID="cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0" exitCode=0
Jul 09 18:48:53.165495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.165362    2558 generic.go:356] "Generic (PLEG): container finished" podID="3990a744-136c-4cee-89ca-454c73680476" containerID="9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4" exitCode=0
Jul 09 18:48:53.165495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.165386    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" event={"ID":"3990a744-136c-4cee-89ca-454c73680476","Type":"ContainerDied","Data":"cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0"}
Jul 09 18:48:53.165495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.165439    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" event={"ID":"3990a744-136c-4cee-89ca-454c73680476","Type":"ContainerDied","Data":"9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4"}
Jul 09 18:48:53.165495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.165462    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k" event={"ID":"3990a744-136c-4cee-89ca-454c73680476","Type":"ContainerDied","Data":"c8f32f4f2b3179e9f42e6c4be7f4f59c73def0563a33ccba487fb9b226a1d2b5"}
Jul 09 18:48:53.165495 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.165391    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"
Jul 09 18:48:53.174154 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.174135    2558 scope.go:117] "RemoveContainer" containerID="f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3"
Jul 09 18:48:53.182355 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.182317    2558 scope.go:117] "RemoveContainer" containerID="16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8"
Jul 09 18:48:53.182641 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:48:53.182619    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8\": container with ID starting with 16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8 not found: ID does not exist" containerID="16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8"
Jul 09 18:48:53.182729 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.182711    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8"} err="failed to get container status \"16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8\": rpc error: code = NotFound desc = could not find container \"16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8\": container with ID starting with 16aff41279e62dbc1f049f85815062c73fbdbf7aa332c508418f3836e3d4b7b8 not found: ID does not exist"
Jul 09 18:48:53.182771 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.182731    2558 scope.go:117] "RemoveContainer" containerID="f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3"
Jul 09 18:48:53.183015 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:48:53.182987    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3\": container with ID starting with f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3 not found: ID does not exist" containerID="f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3"
Jul 09 18:48:53.183074 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.183027    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3"} err="failed to get container status \"f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3\": rpc error: code = NotFound desc = could not find container \"f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3\": container with ID starting with f2e41af86fc0ed2d91d839599866c215c411ef5c5e09537fe8fa720f989714c3 not found: ID does not exist"
Jul 09 18:48:53.183074 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.183051    2558 scope.go:117] "RemoveContainer" containerID="cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0"
Jul 09 18:48:53.191854 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.191838    2558 scope.go:117] "RemoveContainer" containerID="6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0"
Jul 09 18:48:53.199471 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.199456    2558 scope.go:117] "RemoveContainer" containerID="9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4"
Jul 09 18:48:53.206286 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.206270    2558 scope.go:117] "RemoveContainer" containerID="cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0"
Jul 09 18:48:53.206618 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:48:53.206595    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0\": container with ID starting with cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0 not found: ID does not exist" containerID="cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0"
Jul 09 18:48:53.206719 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.206629    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0"} err="failed to get container status \"cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0\": rpc error: code = NotFound desc = could not find container \"cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0\": container with ID starting with cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0 not found: ID does not exist"
Jul 09 18:48:53.206719 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.206652    2558 scope.go:117] "RemoveContainer" containerID="6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0"
Jul 09 18:48:53.206989 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:48:53.206964    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0\": container with ID starting with 6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0 not found: ID does not exist" containerID="6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0"
Jul 09 18:48:53.207090 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.207000    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0"} err="failed to get container status \"6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0\": rpc error: code = NotFound desc = could not find container \"6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0\": container with ID starting with 6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0 not found: ID does not exist"
Jul 09 18:48:53.207090 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.207023    2558 scope.go:117] "RemoveContainer" containerID="9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4"
Jul 09 18:48:53.207413 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:48:53.207387    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4\": container with ID starting with 9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4 not found: ID does not exist" containerID="9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4"
Jul 09 18:48:53.207503 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.207419    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4"} err="failed to get container status \"9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4\": rpc error: code = NotFound desc = could not find container \"9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4\": container with ID starting with 9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4 not found: ID does not exist"
Jul 09 18:48:53.207503 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.207442    2558 scope.go:117] "RemoveContainer" containerID="cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0"
Jul 09 18:48:53.208123 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.208025    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0"} err="failed to get container status \"cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0\": rpc error: code = NotFound desc = could not find container \"cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0\": container with ID starting with cd81847f9fac22d2d7ee4496cf0b0df5f673b5eb0c8cd390b711a62f63b839b0 not found: ID does not exist"
Jul 09 18:48:53.208123 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.208053    2558 scope.go:117] "RemoveContainer" containerID="6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0"
Jul 09 18:48:53.208416 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.208394    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0"} err="failed to get container status \"6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0\": rpc error: code = NotFound desc = could not find container \"6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0\": container with ID starting with 6a0e8b7988282fd808f59b7e1ba1a82354de908a445dcbce312306d139471bf0 not found: ID does not exist"
Jul 09 18:48:53.208504 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.208418    2558 scope.go:117] "RemoveContainer" containerID="9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4"
Jul 09 18:48:53.208689 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.208670    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4"} err="failed to get container status \"9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4\": rpc error: code = NotFound desc = could not find container \"9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4\": container with ID starting with 9ab04d4912088dae9ae277700deb1230a52ec21fd9596eb1ffbdb8fe16019aa4 not found: ID does not exist"
Jul 09 18:48:53.210251 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.210231    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"]
Jul 09 18:48:53.218701 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.218679    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d4dd88bc4-5jk4k"]
Jul 09 18:48:53.238739 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.238710    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"]
Jul 09 18:48:53.243640 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:53.243620    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6d9fbb969f-x899s"]
Jul 09 18:48:54.893270 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:54.893241    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3990a744-136c-4cee-89ca-454c73680476" path="/var/lib/kubelet/pods/3990a744-136c-4cee-89ca-454c73680476/volumes"
Jul 09 18:48:54.893761 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:54.893747    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" path="/var/lib/kubelet/pods/99b4273c-4846-409c-ab22-a1c3a9516de2/volumes"
Jul 09 18:48:55.190837 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:48:55.190800    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 18:49:00.875819 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.875778    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"]
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876082    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="storage-initializer"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876095    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="storage-initializer"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876106    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876112    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876119    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="storage-initializer"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876125    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="storage-initializer"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876138    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="llm-d-routing-sidecar"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876143    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="llm-d-routing-sidecar"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876151    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876156    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main"
Jul 09 18:49:00.876200 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876201    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="llm-d-routing-sidecar"
Jul 09 18:49:00.876576 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876209    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b4273c-4846-409c-ab22-a1c3a9516de2" containerName="main"
Jul 09 18:49:00.876576 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.876215    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3990a744-136c-4cee-89ca-454c73680476" containerName="main"
Jul 09 18:49:00.879487 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.879467    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:00.882965 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.882944    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 09 18:49:00.893834 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.893811    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"]
Jul 09 18:49:00.978024 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.977989    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f343785-618d-4857-8cc4-bc619c9515ee-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:00.978024 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.978040    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:00.978284 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.978115    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:00.978284 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.978163    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:00.978284 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.978206    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:00.978466 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.978379    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:00.978466 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:00.978423    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwbg\" (UniqueName: \"kubernetes.io/projected/6f343785-618d-4857-8cc4-bc619c9515ee-kube-api-access-dzwbg\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.079764 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.079727    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f343785-618d-4857-8cc4-bc619c9515ee-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.079929 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.079773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.079929 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.079815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.079929 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.079842    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.079929 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.079877    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.080116 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.079938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.080116 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.079966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwbg\" (UniqueName: \"kubernetes.io/projected/6f343785-618d-4857-8cc4-bc619c9515ee-kube-api-access-dzwbg\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.080468 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.080434    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.080603 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.080480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.080603 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.080556    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.080603 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.080593    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.082251 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.082232    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.082800 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.082777    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f343785-618d-4857-8cc4-bc619c9515ee-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.089650 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.089626    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwbg\" (UniqueName: \"kubernetes.io/projected/6f343785-618d-4857-8cc4-bc619c9515ee-kube-api-access-dzwbg\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.190144 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.190114    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:01.341433 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:01.341354    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"]
Jul 09 18:49:01.344034 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:49:01.344004    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f343785_618d_4857_8cc4_bc619c9515ee.slice/crio-b4582abcf31d99f3b8a4f8300376c84609b510de1e9aaed9f95ee97285151f34 WatchSource:0}: Error finding container b4582abcf31d99f3b8a4f8300376c84609b510de1e9aaed9f95ee97285151f34: Status 404 returned error can't find the container with id b4582abcf31d99f3b8a4f8300376c84609b510de1e9aaed9f95ee97285151f34
Jul 09 18:49:02.208376 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:02.208342    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" event={"ID":"6f343785-618d-4857-8cc4-bc619c9515ee","Type":"ContainerStarted","Data":"b11d879fd6a6daf2e4d41ac7647baa0467842f6002c4134602657957c6692352"}
Jul 09 18:49:02.208788 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:02.208397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" event={"ID":"6f343785-618d-4857-8cc4-bc619c9515ee","Type":"ContainerStarted","Data":"b4582abcf31d99f3b8a4f8300376c84609b510de1e9aaed9f95ee97285151f34"}
Jul 09 18:49:05.191198 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:05.191143    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 18:49:05.228391 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:05.228352    2558 generic.go:356] "Generic (PLEG): container finished" podID="6f343785-618d-4857-8cc4-bc619c9515ee" containerID="b11d879fd6a6daf2e4d41ac7647baa0467842f6002c4134602657957c6692352" exitCode=0
Jul 09 18:49:05.228569 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:05.228416    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" event={"ID":"6f343785-618d-4857-8cc4-bc619c9515ee","Type":"ContainerDied","Data":"b11d879fd6a6daf2e4d41ac7647baa0467842f6002c4134602657957c6692352"}
Jul 09 18:49:06.235115 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:06.235075    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" event={"ID":"6f343785-618d-4857-8cc4-bc619c9515ee","Type":"ContainerStarted","Data":"2324eff44945d05e0d75fe5865ef3248d144be3e294bd6ec513098e4105fdccb"}
Jul 09 18:49:06.270521 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:06.270470    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podStartSLOduration=2.999662358 podStartE2EDuration="6.270451806s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.270789448s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:49:06.268285543 +0000 UTC m=+2741.914279081" watchObservedRunningTime="2026-07-09 18:49:06.270451806 +0000 UTC m=+2741.916445385"
Jul 09 18:49:07.190726 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:07.190688    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:11.191077 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:11.191030    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:49:11.192402 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:11.192374    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:49:15.191165 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:15.191116    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 18:49:21.190899 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:21.190853    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:49:25.191083 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:25.191039    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 18:49:31.191341 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:31.191288    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:49:35.191199 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:35.191147    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 18:49:41.191270 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:41.191219    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:49:45.202063 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:45.202016    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:49:45.210886 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:45.210851    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:49:51.191229 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:49:51.191181    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:50:01.191396 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:01.191301    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:50:11.191040 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:11.190980    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:50:21.191453 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:21.191397    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:50:23.239154 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:23.239121    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"]
Jul 09 18:50:31.190596 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:31.190547    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:50:38.262446 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:38.262401    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" containerID="cri-o://e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80" gracePeriod=45
Jul 09 18:50:39.229609 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.229545    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:50:39.375481 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375445    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-cabundle-cert\") pod \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") "
Jul 09 18:50:39.375947 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375505    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tls-certs\") pod \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") "
Jul 09 18:50:39.375947 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375547    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-home\") pod \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") "
Jul 09 18:50:39.375947 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375586    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-dshm\") pod \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") "
Jul 09 18:50:39.375947 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375726    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-model-cache\") pod \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") "
Jul 09 18:50:39.375947 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375793    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx9qb\" (UniqueName: \"kubernetes.io/projected/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kube-api-access-qx9qb\") pod \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") "
Jul 09 18:50:39.375947 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375837    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tmp-dir\") pod \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") "
Jul 09 18:50:39.375947 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375871    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kserve-provision-location\") pod \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\" (UID: \"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb\") "
Jul 09 18:50:39.375947 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375871    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" (UID: "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:50:39.376575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.375954    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-home" (OuterVolumeSpecName: "home") pod "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" (UID: "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:50:39.376575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.376255    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:50:39.376575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.376278    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-cabundle-cert\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:50:39.376575 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.376372    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" (UID: "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:50:39.378060 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.378014    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" (UID: "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:50:39.378389 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.378368    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-dshm" (OuterVolumeSpecName: "dshm") pod "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" (UID: "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:50:39.378567 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.378552    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kube-api-access-qx9qb" (OuterVolumeSpecName: "kube-api-access-qx9qb") pod "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" (UID: "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb"). InnerVolumeSpecName "kube-api-access-qx9qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:50:39.387616 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.387584    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" (UID: "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:50:39.459205 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.459166    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" (UID: "ca92f8bd-e3e1-41de-bfd3-b926a64e7afb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:50:39.477417 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.477393    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:50:39.477417 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.477418    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:50:39.477417 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.477431    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx9qb\" (UniqueName: \"kubernetes.io/projected/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kube-api-access-qx9qb\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:50:39.477600 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.477439    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:50:39.477600 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.477449    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:50:39.477600 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.477456    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:50:39.707469 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.707431    2558 generic.go:356] "Generic (PLEG): container finished" podID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerID="e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80" exitCode=0
Jul 09 18:50:39.707648 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.707483    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"
Jul 09 18:50:39.707648 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.707531    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" event={"ID":"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb","Type":"ContainerDied","Data":"e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80"}
Jul 09 18:50:39.707648 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.707606    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" event={"ID":"ca92f8bd-e3e1-41de-bfd3-b926a64e7afb","Type":"ContainerDied","Data":"080d21af41d11fd151a9a2f8968b98ffb2b98573cfdd0eb905e3c243db290e3b"}
Jul 09 18:50:39.707648 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.707640    2558 scope.go:117] "RemoveContainer" containerID="e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80"
Jul 09 18:50:39.717035 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.717014    2558 scope.go:117] "RemoveContainer" containerID="aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681"
Jul 09 18:50:39.727220 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.727200    2558 scope.go:117] "RemoveContainer" containerID="e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80"
Jul 09 18:50:39.727583 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:50:39.727559    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80\": container with ID starting with e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80 not found: ID does not exist" containerID="e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80"
Jul 09 18:50:39.727635 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.727615    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80"} err="failed to get container status \"e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80\": rpc error: code = NotFound desc = could not find container \"e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80\": container with ID starting with e8d28f3e483657ad71b226c34015a3360e263f97ca29963fb83322c95a488b80 not found: ID does not exist"
Jul 09 18:50:39.727682 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.727640    2558 scope.go:117] "RemoveContainer" containerID="aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681"
Jul 09 18:50:39.727907 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:50:39.727891    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681\": container with ID starting with aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681 not found: ID does not exist" containerID="aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681"
Jul 09 18:50:39.727949 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.727914    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681"} err="failed to get container status \"aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681\": rpc error: code = NotFound desc = could not find container \"aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681\": container with ID starting with aba3a4dd770bb8c33246cce0f01717c9e6b285e14c16b10d7528f60970fe9681 not found: ID does not exist"
Jul 09 18:50:39.731911 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.731886    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"]
Jul 09 18:50:39.734992 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:39.734963    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb"]
Jul 09 18:50:40.203907 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:40.203864    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-78f49d77c4-4brdb" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 09 18:50:40.893001 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:40.892965    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" path="/var/lib/kubelet/pods/ca92f8bd-e3e1-41de-bfd3-b926a64e7afb/volumes"
Jul 09 18:50:41.191250 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:41.191212    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:50:51.191052 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:51.191007    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:50:58.567948 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.567914    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"]
Jul 09 18:50:58.568385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.568210    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="storage-initializer"
Jul 09 18:50:58.568385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.568220    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="storage-initializer"
Jul 09 18:50:58.568385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.568230    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main"
Jul 09 18:50:58.568385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.568236    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main"
Jul 09 18:50:58.568385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.568288    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca92f8bd-e3e1-41de-bfd3-b926a64e7afb" containerName="main"
Jul 09 18:50:58.574222 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.574204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.577570 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.577532    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:50:58.577727 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.577582    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 09 18:50:58.584559 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.584530    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"]
Jul 09 18:50:58.738801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.738770    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.738801 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.738809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909629e8-54de-4cdd-979d-edba0a89bfac-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.739047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.738837    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-model-cache\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.739047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.738890    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.739047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.738931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-home\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.739047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.738948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjmf\" (UniqueName: \"kubernetes.io/projected/909629e8-54de-4cdd-979d-edba0a89bfac-kube-api-access-kdjmf\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.739047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.738972    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-dshm\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.739047 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.739040    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909629e8-54de-4cdd-979d-edba0a89bfac-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.840479 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909629e8-54de-4cdd-979d-edba0a89bfac-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.840479 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.840685 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909629e8-54de-4cdd-979d-edba0a89bfac-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.840685 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-model-cache\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.840685 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.840921 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-home\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.840921 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840735    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjmf\" (UniqueName: \"kubernetes.io/projected/909629e8-54de-4cdd-979d-edba0a89bfac-kube-api-access-kdjmf\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.840921 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840765    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-dshm\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.840921 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840826    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.841132 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.840922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-model-cache\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.841132 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.841036    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.841132 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.841064    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-home\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.841376 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.841355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909629e8-54de-4cdd-979d-edba0a89bfac-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.843044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.843021    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-dshm\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.843310 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.843293    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909629e8-54de-4cdd-979d-edba0a89bfac-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.850221 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.850197    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjmf\" (UniqueName: \"kubernetes.io/projected/909629e8-54de-4cdd-979d-edba0a89bfac-kube-api-access-kdjmf\") pod \"lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:58.885063 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:58.885034    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:50:59.062157 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:59.062124    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"]
Jul 09 18:50:59.065615 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:50:59.065575    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909629e8_54de_4cdd_979d_edba0a89bfac.slice/crio-7ae4a5c3b9c4bffc8f366bd487460aed4410e92823c271b5055f229755297032 WatchSource:0}: Error finding container 7ae4a5c3b9c4bffc8f366bd487460aed4410e92823c271b5055f229755297032: Status 404 returned error can't find the container with id 7ae4a5c3b9c4bffc8f366bd487460aed4410e92823c271b5055f229755297032
Jul 09 18:50:59.801538 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:59.801483    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" event={"ID":"909629e8-54de-4cdd-979d-edba0a89bfac","Type":"ContainerStarted","Data":"d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea"}
Jul 09 18:50:59.802105 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:50:59.801588    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" event={"ID":"909629e8-54de-4cdd-979d-edba0a89bfac","Type":"ContainerStarted","Data":"7ae4a5c3b9c4bffc8f366bd487460aed4410e92823c271b5055f229755297032"}
Jul 09 18:51:01.191479 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:01.191435    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 18:51:03.820662 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:03.820626    2558 generic.go:356] "Generic (PLEG): container finished" podID="909629e8-54de-4cdd-979d-edba0a89bfac" containerID="d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea" exitCode=0
Jul 09 18:51:03.821072 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:03.820693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" event={"ID":"909629e8-54de-4cdd-979d-edba0a89bfac","Type":"ContainerDied","Data":"d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea"}
Jul 09 18:51:04.827246 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:04.827213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" event={"ID":"909629e8-54de-4cdd-979d-edba0a89bfac","Type":"ContainerStarted","Data":"866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b"}
Jul 09 18:51:04.856409 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:04.856354    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podStartSLOduration=2.830009172 podStartE2EDuration="6.856320191s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.026311019s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:51:04.854975797 +0000 UTC m=+2860.500969339" watchObservedRunningTime="2026-07-09 18:51:04.856320191 +0000 UTC m=+2860.502313732"
Jul 09 18:51:04.885451 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:04.885408    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:51:08.885321 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:08.885279    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:51:08.886893 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:08.886864    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 09 18:51:11.201436 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:11.201398    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:51:11.209965 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:11.209937    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:51:18.885991 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:18.885858    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 09 18:51:28.247000 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:28.246964    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"]
Jul 09 18:51:28.885882 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:28.885828    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 09 18:51:38.885838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:38.885789    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 09 18:51:43.270150 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:43.270076    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main" containerID="cri-o://2324eff44945d05e0d75fe5865ef3248d144be3e294bd6ec513098e4105fdccb" gracePeriod=45
Jul 09 18:51:44.006680 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.006639    2558 generic.go:356] "Generic (PLEG): container finished" podID="6f343785-618d-4857-8cc4-bc619c9515ee" containerID="2324eff44945d05e0d75fe5865ef3248d144be3e294bd6ec513098e4105fdccb" exitCode=0
Jul 09 18:51:44.006873 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.006718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" event={"ID":"6f343785-618d-4857-8cc4-bc619c9515ee","Type":"ContainerDied","Data":"2324eff44945d05e0d75fe5865ef3248d144be3e294bd6ec513098e4105fdccb"}
Jul 09 18:51:44.034315 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.034284    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:51:44.142672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.142139    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-kserve-provision-location\") pod \"6f343785-618d-4857-8cc4-bc619c9515ee\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") "
Jul 09 18:51:44.142672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.142244    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzwbg\" (UniqueName: \"kubernetes.io/projected/6f343785-618d-4857-8cc4-bc619c9515ee-kube-api-access-dzwbg\") pod \"6f343785-618d-4857-8cc4-bc619c9515ee\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") "
Jul 09 18:51:44.142672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.142366    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-tmp-dir\") pod \"6f343785-618d-4857-8cc4-bc619c9515ee\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") "
Jul 09 18:51:44.142672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.142414    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-dshm\") pod \"6f343785-618d-4857-8cc4-bc619c9515ee\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") "
Jul 09 18:51:44.142672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.142452    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f343785-618d-4857-8cc4-bc619c9515ee-tls-certs\") pod \"6f343785-618d-4857-8cc4-bc619c9515ee\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") "
Jul 09 18:51:44.142672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.142480    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-home\") pod \"6f343785-618d-4857-8cc4-bc619c9515ee\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") "
Jul 09 18:51:44.142672 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.142528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-model-cache\") pod \"6f343785-618d-4857-8cc4-bc619c9515ee\" (UID: \"6f343785-618d-4857-8cc4-bc619c9515ee\") "
Jul 09 18:51:44.143186 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.143127    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f343785-618d-4857-8cc4-bc619c9515ee" (UID: "6f343785-618d-4857-8cc4-bc619c9515ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:51:44.149813 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.149771    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-home" (OuterVolumeSpecName: "home") pod "6f343785-618d-4857-8cc4-bc619c9515ee" (UID: "6f343785-618d-4857-8cc4-bc619c9515ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:51:44.152360 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.152277    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-dshm" (OuterVolumeSpecName: "dshm") pod "6f343785-618d-4857-8cc4-bc619c9515ee" (UID: "6f343785-618d-4857-8cc4-bc619c9515ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:51:44.152716 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.152678    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f343785-618d-4857-8cc4-bc619c9515ee-kube-api-access-dzwbg" (OuterVolumeSpecName: "kube-api-access-dzwbg") pod "6f343785-618d-4857-8cc4-bc619c9515ee" (UID: "6f343785-618d-4857-8cc4-bc619c9515ee"). InnerVolumeSpecName "kube-api-access-dzwbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:51:44.152824 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.152713    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f343785-618d-4857-8cc4-bc619c9515ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f343785-618d-4857-8cc4-bc619c9515ee" (UID: "6f343785-618d-4857-8cc4-bc619c9515ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:51:44.162820 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.162796    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f343785-618d-4857-8cc4-bc619c9515ee" (UID: "6f343785-618d-4857-8cc4-bc619c9515ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:51:44.211167 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.211128    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f343785-618d-4857-8cc4-bc619c9515ee" (UID: "6f343785-618d-4857-8cc4-bc619c9515ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:51:44.243759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.243726    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:51:44.243759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.243757    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:51:44.243759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.243767    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwbg\" (UniqueName: \"kubernetes.io/projected/6f343785-618d-4857-8cc4-bc619c9515ee-kube-api-access-dzwbg\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:51:44.244081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.243776    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:51:44.244081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.243787    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:51:44.244081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.243794    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f343785-618d-4857-8cc4-bc619c9515ee-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:51:44.244081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:44.243801    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f343785-618d-4857-8cc4-bc619c9515ee-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:51:45.014643 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:45.014595    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc" event={"ID":"6f343785-618d-4857-8cc4-bc619c9515ee","Type":"ContainerDied","Data":"b4582abcf31d99f3b8a4f8300376c84609b510de1e9aaed9f95ee97285151f34"}
Jul 09 18:51:45.015098 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:45.014649    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"
Jul 09 18:51:45.015098 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:45.014678    2558 scope.go:117] "RemoveContainer" containerID="2324eff44945d05e0d75fe5865ef3248d144be3e294bd6ec513098e4105fdccb"
Jul 09 18:51:45.023527 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:45.023484    2558 scope.go:117] "RemoveContainer" containerID="b11d879fd6a6daf2e4d41ac7647baa0467842f6002c4134602657957c6692352"
Jul 09 18:51:45.036184 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:45.036159    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"]
Jul 09 18:51:45.040175 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:45.040149    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-76569c86d4h4llc"]
Jul 09 18:51:46.892785 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:46.892755    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" path="/var/lib/kubelet/pods/6f343785-618d-4857-8cc4-bc619c9515ee/volumes"
Jul 09 18:51:48.885732 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:48.885685    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 09 18:51:53.526650 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.526616    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:51:53.527253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.526907    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="storage-initializer"
Jul 09 18:51:53.527253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.526918    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="storage-initializer"
Jul 09 18:51:53.527253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.526936    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main"
Jul 09 18:51:53.527253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.526941    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main"
Jul 09 18:51:53.527253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.526991    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f343785-618d-4857-8cc4-bc619c9515ee" containerName="main"
Jul 09 18:51:53.530457 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.530434    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.533780 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.533760    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 09 18:51:53.533917 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.533889    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-4l8zc\""
Jul 09 18:51:53.545606 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.545584    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:51:53.613756 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.613717    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.613948 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.613787    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.613948 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.613809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frnrl\" (UniqueName: \"kubernetes.io/projected/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kube-api-access-frnrl\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.613948 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.613866    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.613948 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.613882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.614125 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.613957    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.614125 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.613976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.714642 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.714605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.714838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.714662    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.714838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.714690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frnrl\" (UniqueName: \"kubernetes.io/projected/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kube-api-access-frnrl\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.714838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.714729    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.714838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.714751    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.714838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.714783    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.714838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.714807    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.715232 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.715084    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.715232 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.715165    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.715232 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.715188    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.715394 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.715282    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.717461 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.717439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.717566 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.717498    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.731429 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.731399    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frnrl\" (UniqueName: \"kubernetes.io/projected/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kube-api-access-frnrl\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.840493 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.840407    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:51:53.991835 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:53.990925    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:51:53.996400 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:51:53.996366    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac427ed6_b7b5_49b4_b2d3_a4e3f0ad8c61.slice/crio-62f3c99d3dd95986c03520fec9f0984715ab3eb2fa747a8daae74108d4632012 WatchSource:0}: Error finding container 62f3c99d3dd95986c03520fec9f0984715ab3eb2fa747a8daae74108d4632012: Status 404 returned error can't find the container with id 62f3c99d3dd95986c03520fec9f0984715ab3eb2fa747a8daae74108d4632012
Jul 09 18:51:54.057960 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:54.057929    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61","Type":"ContainerStarted","Data":"62f3c99d3dd95986c03520fec9f0984715ab3eb2fa747a8daae74108d4632012"}
Jul 09 18:51:55.064438 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:55.064396    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61","Type":"ContainerStarted","Data":"9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89"}
Jul 09 18:51:58.080912 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:58.080878    2558 generic.go:356] "Generic (PLEG): container finished" podID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerID="9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89" exitCode=0
Jul 09 18:51:58.081321 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:58.080959    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61","Type":"ContainerDied","Data":"9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89"}
Jul 09 18:51:58.886646 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:58.886587    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 09 18:51:59.088615 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:59.088570    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61","Type":"ContainerStarted","Data":"2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49"}
Jul 09 18:51:59.117695 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:59.117647    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.022067501 podStartE2EDuration="6.117632219s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.095564718s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:51:59.115372792 +0000 UTC m=+2914.761366352" watchObservedRunningTime="2026-07-09 18:51:59.117632219 +0000 UTC m=+2914.763625757"
Jul 09 18:51:59.840665 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:51:59.840617    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:52:03.840860 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:03.840820    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:52:03.842670 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:03.842635    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:52:08.886594 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:08.886545    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 09 18:52:13.841899 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:13.841842    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:52:18.885758 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:18.885720    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 09 18:52:23.841210 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:23.841163    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:52:28.899261 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:28.899226    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:52:29.252404 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:29.252362    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:52:33.840914 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:33.840857    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:52:43.840877 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:43.840833    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:52:53.841098 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:52:53.841009    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:53:03.841378 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:03.841308    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:53:13.841222 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:13.841173    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:53:19.322109 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:19.322077    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"]
Jul 09 18:53:23.841906 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:23.841860    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:53:25.006414 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:25.006385    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:53:25.013018 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:25.012991    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:53:33.841795 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:33.841743    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:53:34.345856 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:34.345817    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" containerID="cri-o://866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b" gracePeriod=45
Jul 09 18:53:35.295210 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.295183    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:53:35.319784 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.319751    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909629e8-54de-4cdd-979d-edba0a89bfac-cabundle-cert\") pod \"909629e8-54de-4cdd-979d-edba0a89bfac\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") "
Jul 09 18:53:35.319958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.319793    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-tmp-dir\") pod \"909629e8-54de-4cdd-979d-edba0a89bfac\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") "
Jul 09 18:53:35.319958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.319810    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-home\") pod \"909629e8-54de-4cdd-979d-edba0a89bfac\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") "
Jul 09 18:53:35.319958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.319835    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdjmf\" (UniqueName: \"kubernetes.io/projected/909629e8-54de-4cdd-979d-edba0a89bfac-kube-api-access-kdjmf\") pod \"909629e8-54de-4cdd-979d-edba0a89bfac\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") "
Jul 09 18:53:35.320126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.319958    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-dshm\") pod \"909629e8-54de-4cdd-979d-edba0a89bfac\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") "
Jul 09 18:53:35.320126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.320011    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-kserve-provision-location\") pod \"909629e8-54de-4cdd-979d-edba0a89bfac\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") "
Jul 09 18:53:35.320126 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.320038    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-model-cache\") pod \"909629e8-54de-4cdd-979d-edba0a89bfac\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") "
Jul 09 18:53:35.320305 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.320177    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909629e8-54de-4cdd-979d-edba0a89bfac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "909629e8-54de-4cdd-979d-edba0a89bfac" (UID: "909629e8-54de-4cdd-979d-edba0a89bfac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:53:35.320432 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.320408    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-home" (OuterVolumeSpecName: "home") pod "909629e8-54de-4cdd-979d-edba0a89bfac" (UID: "909629e8-54de-4cdd-979d-edba0a89bfac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:53:35.320513 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.320483    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-model-cache" (OuterVolumeSpecName: "model-cache") pod "909629e8-54de-4cdd-979d-edba0a89bfac" (UID: "909629e8-54de-4cdd-979d-edba0a89bfac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:53:35.322838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.322510    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909629e8-54de-4cdd-979d-edba0a89bfac-kube-api-access-kdjmf" (OuterVolumeSpecName: "kube-api-access-kdjmf") pod "909629e8-54de-4cdd-979d-edba0a89bfac" (UID: "909629e8-54de-4cdd-979d-edba0a89bfac"). InnerVolumeSpecName "kube-api-access-kdjmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:53:35.322838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.322586    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-dshm" (OuterVolumeSpecName: "dshm") pod "909629e8-54de-4cdd-979d-edba0a89bfac" (UID: "909629e8-54de-4cdd-979d-edba0a89bfac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:53:35.338163 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.338132    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "909629e8-54de-4cdd-979d-edba0a89bfac" (UID: "909629e8-54de-4cdd-979d-edba0a89bfac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:53:35.390063 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.389973    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "909629e8-54de-4cdd-979d-edba0a89bfac" (UID: "909629e8-54de-4cdd-979d-edba0a89bfac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:53:35.420878 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.420848    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909629e8-54de-4cdd-979d-edba0a89bfac-tls-certs\") pod \"909629e8-54de-4cdd-979d-edba0a89bfac\" (UID: \"909629e8-54de-4cdd-979d-edba0a89bfac\") "
Jul 09 18:53:35.421036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.420988    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909629e8-54de-4cdd-979d-edba0a89bfac-cabundle-cert\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:53:35.421036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.421000    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:53:35.421036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.421008    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:53:35.421036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.421017    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjmf\" (UniqueName: \"kubernetes.io/projected/909629e8-54de-4cdd-979d-edba0a89bfac-kube-api-access-kdjmf\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:53:35.421036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.421025    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:53:35.421036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.421033    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:53:35.421253 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.421041    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909629e8-54de-4cdd-979d-edba0a89bfac-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:53:35.422959 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.422936    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909629e8-54de-4cdd-979d-edba0a89bfac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "909629e8-54de-4cdd-979d-edba0a89bfac" (UID: "909629e8-54de-4cdd-979d-edba0a89bfac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:53:35.522165 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.522130    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909629e8-54de-4cdd-979d-edba0a89bfac-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:53:35.592726 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.592687    2558 generic.go:356] "Generic (PLEG): container finished" podID="909629e8-54de-4cdd-979d-edba0a89bfac" containerID="866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b" exitCode=0
Jul 09 18:53:35.592922 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.592741    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"
Jul 09 18:53:35.592922 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.592768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" event={"ID":"909629e8-54de-4cdd-979d-edba0a89bfac","Type":"ContainerDied","Data":"866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b"}
Jul 09 18:53:35.592922 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.592829    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" event={"ID":"909629e8-54de-4cdd-979d-edba0a89bfac","Type":"ContainerDied","Data":"7ae4a5c3b9c4bffc8f366bd487460aed4410e92823c271b5055f229755297032"}
Jul 09 18:53:35.592922 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.592858    2558 scope.go:117] "RemoveContainer" containerID="866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b"
Jul 09 18:53:35.601770 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.601752    2558 scope.go:117] "RemoveContainer" containerID="d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea"
Jul 09 18:53:35.609910 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.609890    2558 scope.go:117] "RemoveContainer" containerID="866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b"
Jul 09 18:53:35.610171 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:53:35.610154    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b\": container with ID starting with 866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b not found: ID does not exist" containerID="866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b"
Jul 09 18:53:35.610220 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.610180    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b"} err="failed to get container status \"866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b\": rpc error: code = NotFound desc = could not find container \"866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b\": container with ID starting with 866275c105008792d7943d8c0f9e8d832808ef24f59db4872eb423858eff781b not found: ID does not exist"
Jul 09 18:53:35.610220 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.610199    2558 scope.go:117] "RemoveContainer" containerID="d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea"
Jul 09 18:53:35.610438 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:53:35.610423    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea\": container with ID starting with d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea not found: ID does not exist" containerID="d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea"
Jul 09 18:53:35.610490 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.610445    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea"} err="failed to get container status \"d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea\": rpc error: code = NotFound desc = could not find container \"d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea\": container with ID starting with d1a92bb8e1bbaffd2012d772aed52e96b8546c7489735493c8df714f7a0807ea not found: ID does not exist"
Jul 09 18:53:35.618045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.618021    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"]
Jul 09 18:53:35.621946 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:35.621923    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk"]
Jul 09 18:53:36.244488 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:36.244427    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6854f6cbbc-zjrkk" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Jul 09 18:53:36.892809 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:36.892778    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" path="/var/lib/kubelet/pods/909629e8-54de-4cdd-979d-edba0a89bfac/volumes"
Jul 09 18:53:43.841583 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:43.841534    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:53:48.499797 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.499763    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"]
Jul 09 18:53:48.500165 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.500028    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="storage-initializer"
Jul 09 18:53:48.500165 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.500039    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="storage-initializer"
Jul 09 18:53:48.500165 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.500053    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main"
Jul 09 18:53:48.500165 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.500060    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main"
Jul 09 18:53:48.500165 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.500112    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="909629e8-54de-4cdd-979d-edba0a89bfac" containerName="main"
Jul 09 18:53:48.505633 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.505614    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.510524 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.510499    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 09 18:53:48.519960 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.519939    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"]
Jul 09 18:53:48.526562 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.526535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-home\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.526673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.526576    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4b6123-7ac8-411e-909f-5464228cc819-tls-certs\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.526673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.526597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jks7t\" (UniqueName: \"kubernetes.io/projected/1b4b6123-7ac8-411e-909f-5464228cc819-kube-api-access-jks7t\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.526673 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.526660    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-tmp-dir\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.526799 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.526675    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-kserve-provision-location\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.526799 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.526712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-dshm\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.526799 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.526740    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-model-cache\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.627627 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.627588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4b6123-7ac8-411e-909f-5464228cc819-tls-certs\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.627627 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.627625    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks7t\" (UniqueName: \"kubernetes.io/projected/1b4b6123-7ac8-411e-909f-5464228cc819-kube-api-access-jks7t\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.627845 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.627660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-tmp-dir\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.627845 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.627828    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-kserve-provision-location\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.627927 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.627851    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-dshm\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.627927 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.627873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-model-cache\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.627927 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.627900    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-home\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.628174 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.628150    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-tmp-dir\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.628245 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.628160    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-kserve-provision-location\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.628245 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.628210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-home\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.628546 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.628253    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-model-cache\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.630102 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.630080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-dshm\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.630362 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.630320    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4b6123-7ac8-411e-909f-5464228cc819-tls-certs\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.638391 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.638366    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jks7t\" (UniqueName: \"kubernetes.io/projected/1b4b6123-7ac8-411e-909f-5464228cc819-kube-api-access-jks7t\") pod \"tls-verification-test-kserve-66cd955d55-n7crl\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.815653 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.815556    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:48.959720 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:48.959695    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"]
Jul 09 18:53:48.962211 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:53:48.962181    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4b6123_7ac8_411e_909f_5464228cc819.slice/crio-8e6ab7024255be796c94a4c23ab9e8566a9610d1bbb769343618c60c6cb51162 WatchSource:0}: Error finding container 8e6ab7024255be796c94a4c23ab9e8566a9610d1bbb769343618c60c6cb51162: Status 404 returned error can't find the container with id 8e6ab7024255be796c94a4c23ab9e8566a9610d1bbb769343618c60c6cb51162
Jul 09 18:53:49.660177 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:49.660127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" event={"ID":"1b4b6123-7ac8-411e-909f-5464228cc819","Type":"ContainerStarted","Data":"821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433"}
Jul 09 18:53:49.660662 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:49.660197    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" event={"ID":"1b4b6123-7ac8-411e-909f-5464228cc819","Type":"ContainerStarted","Data":"8e6ab7024255be796c94a4c23ab9e8566a9610d1bbb769343618c60c6cb51162"}
Jul 09 18:53:52.675516 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:52.675427    2558 generic.go:356] "Generic (PLEG): container finished" podID="1b4b6123-7ac8-411e-909f-5464228cc819" containerID="821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433" exitCode=0
Jul 09 18:53:52.675516 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:52.675503    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" event={"ID":"1b4b6123-7ac8-411e-909f-5464228cc819","Type":"ContainerDied","Data":"821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433"}
Jul 09 18:53:53.682838 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:53.682801    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" event={"ID":"1b4b6123-7ac8-411e-909f-5464228cc819","Type":"ContainerStarted","Data":"4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536"}
Jul 09 18:53:53.709418 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:53.709346    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podStartSLOduration=2.318321452 podStartE2EDuration="5.709307209s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.390985757s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:53.706750879 +0000 UTC m=+3029.352744418" watchObservedRunningTime="2026-07-09 18:53:53.709307209 +0000 UTC m=+3029.355300750"
Jul 09 18:53:53.815984 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:53.815947    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:53.851013 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:53.850978    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:53.858878 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:53.858854    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:53:58.815958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:58.815920    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:53:58.817749 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:53:58.817721    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:54:08.816970 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:54:08.816904    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:54:18.816301 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:54:18.816221    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:54:28.816765 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:54:28.816713    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:54:38.816177 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:54:38.816135    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:54:48.816349 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:54:48.816276    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:54:58.816510 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:54:58.816464    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:55:08.817042 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:55:08.816990    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:55:18.816962 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:55:18.816912    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:55:28.816151 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:55:28.816103    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:55:38.816212 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:55:38.816166    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:55:48.826110 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:55:48.826021    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:55:48.833566 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:55:48.833541    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:55:54.256949 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:55:54.256911    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"]
Jul 09 18:56:00.300348 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:00.300303    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:56:09.280693 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:09.280651    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" containerID="cri-o://4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536" gracePeriod=45
Jul 09 18:56:09.826648 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:09.826613    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:56:10.033870 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.033847    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:56:10.163895 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.163808    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jks7t\" (UniqueName: \"kubernetes.io/projected/1b4b6123-7ac8-411e-909f-5464228cc819-kube-api-access-jks7t\") pod \"1b4b6123-7ac8-411e-909f-5464228cc819\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") "
Jul 09 18:56:10.163895 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.163858    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-kserve-provision-location\") pod \"1b4b6123-7ac8-411e-909f-5464228cc819\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") "
Jul 09 18:56:10.163895 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.163887    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-dshm\") pod \"1b4b6123-7ac8-411e-909f-5464228cc819\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") "
Jul 09 18:56:10.164138 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.163981    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-home\") pod \"1b4b6123-7ac8-411e-909f-5464228cc819\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") "
Jul 09 18:56:10.164138 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.164023    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-model-cache\") pod \"1b4b6123-7ac8-411e-909f-5464228cc819\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") "
Jul 09 18:56:10.164138 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.164055    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4b6123-7ac8-411e-909f-5464228cc819-tls-certs\") pod \"1b4b6123-7ac8-411e-909f-5464228cc819\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") "
Jul 09 18:56:10.164138 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.164117    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-tmp-dir\") pod \"1b4b6123-7ac8-411e-909f-5464228cc819\" (UID: \"1b4b6123-7ac8-411e-909f-5464228cc819\") "
Jul 09 18:56:10.164371 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.164269    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b4b6123-7ac8-411e-909f-5464228cc819" (UID: "1b4b6123-7ac8-411e-909f-5464228cc819"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:10.164447 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.164423    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:10.164905 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.164876    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-home" (OuterVolumeSpecName: "home") pod "1b4b6123-7ac8-411e-909f-5464228cc819" (UID: "1b4b6123-7ac8-411e-909f-5464228cc819"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:10.166316 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.166284    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4b6123-7ac8-411e-909f-5464228cc819-kube-api-access-jks7t" (OuterVolumeSpecName: "kube-api-access-jks7t") pod "1b4b6123-7ac8-411e-909f-5464228cc819" (UID: "1b4b6123-7ac8-411e-909f-5464228cc819"). InnerVolumeSpecName "kube-api-access-jks7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:56:10.166479 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.166456    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-dshm" (OuterVolumeSpecName: "dshm") pod "1b4b6123-7ac8-411e-909f-5464228cc819" (UID: "1b4b6123-7ac8-411e-909f-5464228cc819"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:10.166675 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.166646    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4b6123-7ac8-411e-909f-5464228cc819-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b4b6123-7ac8-411e-909f-5464228cc819" (UID: "1b4b6123-7ac8-411e-909f-5464228cc819"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:56:10.176759 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.176731    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b4b6123-7ac8-411e-909f-5464228cc819" (UID: "1b4b6123-7ac8-411e-909f-5464228cc819"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:10.237854 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.237801    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b4b6123-7ac8-411e-909f-5464228cc819" (UID: "1b4b6123-7ac8-411e-909f-5464228cc819"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:10.265405 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.265365    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:10.265405 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.265401    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:10.265405 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.265410    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4b6123-7ac8-411e-909f-5464228cc819-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:10.265405 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.265420    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:10.265657 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.265429    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jks7t\" (UniqueName: \"kubernetes.io/projected/1b4b6123-7ac8-411e-909f-5464228cc819-kube-api-access-jks7t\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:10.265657 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.265438    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4b6123-7ac8-411e-909f-5464228cc819-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:10.294823 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.294795    2558 generic.go:356] "Generic (PLEG): container finished" podID="1b4b6123-7ac8-411e-909f-5464228cc819" containerID="4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536" exitCode=0
Jul 09 18:56:10.295155 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.294856    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"
Jul 09 18:56:10.295155 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.294875    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" event={"ID":"1b4b6123-7ac8-411e-909f-5464228cc819","Type":"ContainerDied","Data":"4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536"}
Jul 09 18:56:10.295155 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.294937    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl" event={"ID":"1b4b6123-7ac8-411e-909f-5464228cc819","Type":"ContainerDied","Data":"8e6ab7024255be796c94a4c23ab9e8566a9610d1bbb769343618c60c6cb51162"}
Jul 09 18:56:10.295155 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.294968    2558 scope.go:117] "RemoveContainer" containerID="4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536"
Jul 09 18:56:10.306571 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.304734    2558 scope.go:117] "RemoveContainer" containerID="821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433"
Jul 09 18:56:10.315958 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.315938    2558 scope.go:117] "RemoveContainer" containerID="4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536"
Jul 09 18:56:10.316209 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:56:10.316185    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536\": container with ID starting with 4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536 not found: ID does not exist" containerID="4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536"
Jul 09 18:56:10.316293 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.316224    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536"} err="failed to get container status \"4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536\": rpc error: code = NotFound desc = could not find container \"4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536\": container with ID starting with 4434a49d31959e72383e7471b2bcbefcf2b436f434f09b5f121cd7749858b536 not found: ID does not exist"
Jul 09 18:56:10.316293 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.316245    2558 scope.go:117] "RemoveContainer" containerID="821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433"
Jul 09 18:56:10.316481 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:56:10.316464    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433\": container with ID starting with 821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433 not found: ID does not exist" containerID="821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433"
Jul 09 18:56:10.316518 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.316487    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433"} err="failed to get container status \"821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433\": rpc error: code = NotFound desc = could not find container \"821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433\": container with ID starting with 821cbc096b2d1684d900609bd4c2476bc53efe98cd1652f3343444a37a1be433 not found: ID does not exist"
Jul 09 18:56:10.322291 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.322270    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"]
Jul 09 18:56:10.329433 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.329409    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66cd955d55-n7crl"]
Jul 09 18:56:10.892536 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:10.892502    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" path="/var/lib/kubelet/pods/1b4b6123-7ac8-411e-909f-5464228cc819/volumes"
Jul 09 18:56:15.323717 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:15.323683    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" containerID="cri-o://2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49" gracePeriod=45
Jul 09 18:56:15.851896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:15.851860    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 18:56:16.089604 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.089552    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:56:16.101400 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.101376    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frnrl\" (UniqueName: \"kubernetes.io/projected/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kube-api-access-frnrl\") pod \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") "
Jul 09 18:56:16.101522 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.101415    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-dshm\") pod \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") "
Jul 09 18:56:16.101522 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.101454    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kserve-provision-location\") pod \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") "
Jul 09 18:56:16.101522 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.101490    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-home\") pod \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") "
Jul 09 18:56:16.101687 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.101540    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tmp-dir\") pod \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") "
Jul 09 18:56:16.101687 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.101566    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-model-cache\") pod \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") "
Jul 09 18:56:16.101687 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.101603    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tls-certs\") pod \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\" (UID: \"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61\") "
Jul 09 18:56:16.101886 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.101858    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" (UID: "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:16.102176 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.102135    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-home" (OuterVolumeSpecName: "home") pod "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" (UID: "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:16.104192 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.104168    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-dshm" (OuterVolumeSpecName: "dshm") pod "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" (UID: "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:16.104192 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.104178    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" (UID: "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:56:16.104396 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.104193    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kube-api-access-frnrl" (OuterVolumeSpecName: "kube-api-access-frnrl") pod "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" (UID: "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61"). InnerVolumeSpecName "kube-api-access-frnrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:56:16.114918 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.114888    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" (UID: "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:16.170275 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.170229    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" (UID: "ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:56:16.202945 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.202911    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:16.202945 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.202940    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frnrl\" (UniqueName: \"kubernetes.io/projected/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kube-api-access-frnrl\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:16.202945 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.202950    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:16.203170 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.202960    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:16.203170 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.202970    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:16.203170 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.202979    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:16.203170 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.202986    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:56:16.323542 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.323498    2558 generic.go:356] "Generic (PLEG): container finished" podID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerID="2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49" exitCode=0
Jul 09 18:56:16.323702 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.323556    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 18:56:16.323702 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.323581    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61","Type":"ContainerDied","Data":"2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49"}
Jul 09 18:56:16.323702 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.323647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61","Type":"ContainerDied","Data":"62f3c99d3dd95986c03520fec9f0984715ab3eb2fa747a8daae74108d4632012"}
Jul 09 18:56:16.323702 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.323669    2558 scope.go:117] "RemoveContainer" containerID="2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49"
Jul 09 18:56:16.336479 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.332702    2558 scope.go:117] "RemoveContainer" containerID="9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89"
Jul 09 18:56:16.344291 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.344267    2558 scope.go:117] "RemoveContainer" containerID="2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49"
Jul 09 18:56:16.344719 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:56:16.344679    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49\": container with ID starting with 2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49 not found: ID does not exist" containerID="2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49"
Jul 09 18:56:16.344782 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.344743    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49"} err="failed to get container status \"2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49\": rpc error: code = NotFound desc = could not find container \"2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49\": container with ID starting with 2f9e1b4a1f2797e713e6b7f90af2137fb5e2936abcabc101d7c656ca5754ff49 not found: ID does not exist"
Jul 09 18:56:16.344827 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.344792    2558 scope.go:117] "RemoveContainer" containerID="9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89"
Jul 09 18:56:16.345131 ip-10-0-140-224 kubenswrapper[2558]: E0709 18:56:16.345109    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89\": container with ID starting with 9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89 not found: ID does not exist" containerID="9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89"
Jul 09 18:56:16.345208 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.345146    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89"} err="failed to get container status \"9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89\": rpc error: code = NotFound desc = could not find container \"9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89\": container with ID starting with 9ac5b017f0d7fd114d2f72eb97ff5246c7e44e3389c887df174aba9ed6508b89 not found: ID does not exist"
Jul 09 18:56:16.352905 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.352877    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:56:16.357183 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.357157    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 18:56:16.893044 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:16.893007    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" path="/var/lib/kubelet/pods/ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61/volumes"
Jul 09 18:56:26.898896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.898862    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"]
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899130    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main"
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899142    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main"
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899161    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main"
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899166    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main"
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899173    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="storage-initializer"
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899179    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="storage-initializer"
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899189    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="storage-initializer"
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899195    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="storage-initializer"
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899240    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac427ed6-b7b5-49b4-b2d3-a4e3f0ad8c61" containerName="main"
Jul 09 18:56:26.899265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.899248    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4b6123-7ac8-411e-909f-5464228cc819" containerName="main"
Jul 09 18:56:26.904098 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.904079    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:26.908662 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.908641    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:56:26.908799 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.908701    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 18:56:26.909490 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.909472    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 09 18:56:26.924882 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.924849    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"]
Jul 09 18:56:26.984879 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.984845    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-dshm\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:26.985038 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.984894    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-tmp-dir\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:26.985038 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.984920    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-model-cache\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:26.985038 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.984938    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsdd\" (UniqueName: \"kubernetes.io/projected/76056b0b-e73b-47e7-93cc-685da9f680c7-kube-api-access-rnsdd\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:26.985038 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.984969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-kserve-provision-location\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:26.985185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.985046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76056b0b-e73b-47e7-93cc-685da9f680c7-tls-certs\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:26.985185 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:26.985082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-home\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086077 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-tmp-dir\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086077 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-model-cache\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086317 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsdd\" (UniqueName: \"kubernetes.io/projected/76056b0b-e73b-47e7-93cc-685da9f680c7-kube-api-access-rnsdd\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086317 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-kserve-provision-location\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086317 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76056b0b-e73b-47e7-93cc-685da9f680c7-tls-certs\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086317 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086178    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-home\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086317 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086225    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-dshm\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086610 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086503    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-tmp-dir\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086610 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-model-cache\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086681 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086609    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-kserve-provision-location\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.086715 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.086685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-home\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.088523 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.088502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-dshm\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.088733 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.088712    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76056b0b-e73b-47e7-93cc-685da9f680c7-tls-certs\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.099136 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.099112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsdd\" (UniqueName: \"kubernetes.io/projected/76056b0b-e73b-47e7-93cc-685da9f680c7-kube-api-access-rnsdd\") pod \"prestop-hook-test-kserve-596c6676c9-5gwxg\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.213381 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.213316    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:27.371037 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:27.371008    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"]
Jul 09 18:56:27.371148 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:56:27.371033    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76056b0b_e73b_47e7_93cc_685da9f680c7.slice/crio-2f2197823a360807c14115782c49a4a5fbf871bd0ac0f257d2aa7b007e2a3520 WatchSource:0}: Error finding container 2f2197823a360807c14115782c49a4a5fbf871bd0ac0f257d2aa7b007e2a3520: Status 404 returned error can't find the container with id 2f2197823a360807c14115782c49a4a5fbf871bd0ac0f257d2aa7b007e2a3520
Jul 09 18:56:28.378510 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:28.378457    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" event={"ID":"76056b0b-e73b-47e7-93cc-685da9f680c7","Type":"ContainerStarted","Data":"b0bf5923526967472bfe4fd782bb43fe7d3e6b952239f3dfde24d791467f823a"}
Jul 09 18:56:28.379007 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:28.378534    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" event={"ID":"76056b0b-e73b-47e7-93cc-685da9f680c7","Type":"ContainerStarted","Data":"2f2197823a360807c14115782c49a4a5fbf871bd0ac0f257d2aa7b007e2a3520"}
Jul 09 18:56:31.393038 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:31.393000    2558 generic.go:356] "Generic (PLEG): container finished" podID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerID="b0bf5923526967472bfe4fd782bb43fe7d3e6b952239f3dfde24d791467f823a" exitCode=0
Jul 09 18:56:31.393489 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:31.393058    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" event={"ID":"76056b0b-e73b-47e7-93cc-685da9f680c7","Type":"ContainerDied","Data":"b0bf5923526967472bfe4fd782bb43fe7d3e6b952239f3dfde24d791467f823a"}
Jul 09 18:56:32.400265 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:32.400222    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" event={"ID":"76056b0b-e73b-47e7-93cc-685da9f680c7","Type":"ContainerStarted","Data":"c32ffaaaeef72db5aa48b8fe14f98af1c71571ef5bf07270acd8c1ceaaeb3519"}
Jul 09 18:56:32.428210 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:32.428147    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podStartSLOduration=3.030374721 podStartE2EDuration="6.428125146s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.397750425s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:56:32.424847396 +0000 UTC m=+3188.070840935" watchObservedRunningTime="2026-07-09 18:56:32.428125146 +0000 UTC m=+3188.074118686"
Jul 09 18:56:33.214374 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:33.214313    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:37.213587 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:37.213555    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:56:37.214754 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:37.214723    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:56:47.213850 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:47.213802    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:56:57.214079 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:56:57.214029    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:57:07.214094 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:57:07.214044    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:57:17.214609 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:57:17.214514    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:57:27.214656 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:57:27.214610    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:57:37.214567 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:57:37.214518    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:57:47.214665 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:57:47.214618    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:57:57.214701 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:57:57.214654    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:58:07.214250 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:07.214208    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:58:17.213846 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:17.213793    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 18:58:25.028674 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:25.028646    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:58:25.035778 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:25.035757    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 18:58:27.224017 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:27.223984    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:58:27.237099 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:27.237071    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:58:30.233350 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.233298    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"]
Jul 09 18:58:30.257002 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.256971    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"]
Jul 09 18:58:30.261679 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.261657    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.274034 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.274009    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"]
Jul 09 18:58:30.313081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.313052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6wpt\" (UniqueName: \"kubernetes.io/projected/86bc83f1-769a-4124-bf77-5e919732b28c-kube-api-access-t6wpt\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.313385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.313145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-tmp-dir\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.313385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.313168    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-dshm\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.313385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.313216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-home\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.313385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.313238    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc83f1-769a-4124-bf77-5e919732b28c-tls-certs\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.313385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.313274    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-model-cache\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.313385 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.313298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-kserve-provision-location\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.413902 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.413875    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-tmp-dir\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414027 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.413907    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-dshm\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414027 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.413937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-home\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414027 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.413954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc83f1-769a-4124-bf77-5e919732b28c-tls-certs\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414027 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.413974    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-model-cache\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414027 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.414000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-kserve-provision-location\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414249 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.414028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6wpt\" (UniqueName: \"kubernetes.io/projected/86bc83f1-769a-4124-bf77-5e919732b28c-kube-api-access-t6wpt\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414384 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.414361    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-tmp-dir\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414465 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.414428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-home\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414517 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.414474    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-kserve-provision-location\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.414517 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.414501    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-model-cache\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.416445 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.416423    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-dshm\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.416625 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.416607    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc83f1-769a-4124-bf77-5e919732b28c-tls-certs\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.424880 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.424857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6wpt\" (UniqueName: \"kubernetes.io/projected/86bc83f1-769a-4124-bf77-5e919732b28c-kube-api-access-t6wpt\") pod \"prestop-hook-test-kserve-596c6676c9-4cbkx\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.571477 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.571409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:30.710125 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.710038    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"]
Jul 09 18:58:30.712250 ip-10-0-140-224 kubenswrapper[2558]: W0709 18:58:30.712218    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bc83f1_769a_4124_bf77_5e919732b28c.slice/crio-1fa8623dfcc86a00943664e029aa9393ceb55eac10bffa0c32efdc43227b9bb1 WatchSource:0}: Error finding container 1fa8623dfcc86a00943664e029aa9393ceb55eac10bffa0c32efdc43227b9bb1: Status 404 returned error can't find the container with id 1fa8623dfcc86a00943664e029aa9393ceb55eac10bffa0c32efdc43227b9bb1
Jul 09 18:58:30.911655 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.911566    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx" event={"ID":"86bc83f1-769a-4124-bf77-5e919732b28c","Type":"ContainerStarted","Data":"0162f6c82cb9a4de42c7ab9df470f3c0372a68f46e4875286682d1543abbf079"}
Jul 09 18:58:30.911655 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:30.911624    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx" event={"ID":"86bc83f1-769a-4124-bf77-5e919732b28c","Type":"ContainerStarted","Data":"1fa8623dfcc86a00943664e029aa9393ceb55eac10bffa0c32efdc43227b9bb1"}
Jul 09 18:58:34.931783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:34.931688    2558 generic.go:356] "Generic (PLEG): container finished" podID="86bc83f1-769a-4124-bf77-5e919732b28c" containerID="0162f6c82cb9a4de42c7ab9df470f3c0372a68f46e4875286682d1543abbf079" exitCode=0
Jul 09 18:58:34.931783 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:34.931729    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx" event={"ID":"86bc83f1-769a-4124-bf77-5e919732b28c","Type":"ContainerDied","Data":"0162f6c82cb9a4de42c7ab9df470f3c0372a68f46e4875286682d1543abbf079"}
Jul 09 18:58:35.938045 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:35.938002    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx" event={"ID":"86bc83f1-769a-4124-bf77-5e919732b28c","Type":"ContainerStarted","Data":"2dc5a8b4c16a743918178a2e26de0547b66fb38e156e481052a248ea05f25030"}
Jul 09 18:58:35.968511 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:35.968449    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx" podStartSLOduration=2.264372698 podStartE2EDuration="5.968427991s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.704055293s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:35.967737431 +0000 UTC m=+3311.613730971" watchObservedRunningTime="2026-07-09 18:58:35.968427991 +0000 UTC m=+3311.614421531"
Jul 09 18:58:36.572034 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:36.571995    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:40.572062 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:40.572025    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:58:40.573574 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:40.573545    2558 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx" podUID="86bc83f1-769a-4124-bf77-5e919732b28c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 18:58:45.256343 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:45.256240    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main" containerID="cri-o://c32ffaaaeef72db5aa48b8fe14f98af1c71571ef5bf07270acd8c1ceaaeb3519" gracePeriod=45
Jul 09 18:58:45.982618 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:45.982584    2558 generic.go:356] "Generic (PLEG): container finished" podID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerID="c32ffaaaeef72db5aa48b8fe14f98af1c71571ef5bf07270acd8c1ceaaeb3519" exitCode=0
Jul 09 18:58:45.982790 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:45.982643    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" event={"ID":"76056b0b-e73b-47e7-93cc-685da9f680c7","Type":"ContainerDied","Data":"c32ffaaaeef72db5aa48b8fe14f98af1c71571ef5bf07270acd8c1ceaaeb3519"}
Jul 09 18:58:46.044433 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.044397    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:58:46.152036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.151961    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76056b0b-e73b-47e7-93cc-685da9f680c7-tls-certs\") pod \"76056b0b-e73b-47e7-93cc-685da9f680c7\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") "
Jul 09 18:58:46.152036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.151999    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-tmp-dir\") pod \"76056b0b-e73b-47e7-93cc-685da9f680c7\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") "
Jul 09 18:58:46.152036 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.152025    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnsdd\" (UniqueName: \"kubernetes.io/projected/76056b0b-e73b-47e7-93cc-685da9f680c7-kube-api-access-rnsdd\") pod \"76056b0b-e73b-47e7-93cc-685da9f680c7\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") "
Jul 09 18:58:46.152314 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.152047    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-kserve-provision-location\") pod \"76056b0b-e73b-47e7-93cc-685da9f680c7\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") "
Jul 09 18:58:46.152314 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.152093    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-model-cache\") pod \"76056b0b-e73b-47e7-93cc-685da9f680c7\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") "
Jul 09 18:58:46.152314 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.152120    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-home\") pod \"76056b0b-e73b-47e7-93cc-685da9f680c7\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") "
Jul 09 18:58:46.152314 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.152150    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-dshm\") pod \"76056b0b-e73b-47e7-93cc-685da9f680c7\" (UID: \"76056b0b-e73b-47e7-93cc-685da9f680c7\") "
Jul 09 18:58:46.152558 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.152401    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "76056b0b-e73b-47e7-93cc-685da9f680c7" (UID: "76056b0b-e73b-47e7-93cc-685da9f680c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:58:46.152847 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.152807    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-home" (OuterVolumeSpecName: "home") pod "76056b0b-e73b-47e7-93cc-685da9f680c7" (UID: "76056b0b-e73b-47e7-93cc-685da9f680c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:58:46.154726 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.154699    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-dshm" (OuterVolumeSpecName: "dshm") pod "76056b0b-e73b-47e7-93cc-685da9f680c7" (UID: "76056b0b-e73b-47e7-93cc-685da9f680c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:58:46.154922 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.154898    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76056b0b-e73b-47e7-93cc-685da9f680c7-kube-api-access-rnsdd" (OuterVolumeSpecName: "kube-api-access-rnsdd") pod "76056b0b-e73b-47e7-93cc-685da9f680c7" (UID: "76056b0b-e73b-47e7-93cc-685da9f680c7"). InnerVolumeSpecName "kube-api-access-rnsdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:58:46.155209 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.155181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76056b0b-e73b-47e7-93cc-685da9f680c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76056b0b-e73b-47e7-93cc-685da9f680c7" (UID: "76056b0b-e73b-47e7-93cc-685da9f680c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:58:46.170837 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.170799    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76056b0b-e73b-47e7-93cc-685da9f680c7" (UID: "76056b0b-e73b-47e7-93cc-685da9f680c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:58:46.225516 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.225466    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76056b0b-e73b-47e7-93cc-685da9f680c7" (UID: "76056b0b-e73b-47e7-93cc-685da9f680c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:58:46.253152 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.253116    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76056b0b-e73b-47e7-93cc-685da9f680c7-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.253152 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.253151    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.253386 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.253161    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsdd\" (UniqueName: \"kubernetes.io/projected/76056b0b-e73b-47e7-93cc-685da9f680c7-kube-api-access-rnsdd\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.253386 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.253172    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.253386 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.253181    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.253386 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.253188    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.253386 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.253195    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76056b0b-e73b-47e7-93cc-685da9f680c7-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:58:46.989120 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.989080    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg" event={"ID":"76056b0b-e73b-47e7-93cc-685da9f680c7","Type":"ContainerDied","Data":"2f2197823a360807c14115782c49a4a5fbf871bd0ac0f257d2aa7b007e2a3520"}
Jul 09 18:58:46.989120 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.989105    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"
Jul 09 18:58:46.989682 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.989164    2558 scope.go:117] "RemoveContainer" containerID="c32ffaaaeef72db5aa48b8fe14f98af1c71571ef5bf07270acd8c1ceaaeb3519"
Jul 09 18:58:46.998081 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:46.998062    2558 scope.go:117] "RemoveContainer" containerID="b0bf5923526967472bfe4fd782bb43fe7d3e6b952239f3dfde24d791467f823a"
Jul 09 18:58:47.019346 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:47.019292    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"]
Jul 09 18:58:47.043514 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:47.043482    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-5gwxg"]
Jul 09 18:58:48.664064 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:48.664019    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"]
Jul 09 18:58:48.894703 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:58:48.894666    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" path="/var/lib/kubelet/pods/76056b0b-e73b-47e7-93cc-685da9f680c7/volumes"
Jul 09 18:59:03.689403 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:03.689357    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx" podUID="86bc83f1-769a-4124-bf77-5e919732b28c" containerName="main" containerID="cri-o://2dc5a8b4c16a743918178a2e26de0547b66fb38e156e481052a248ea05f25030" gracePeriod=45
Jul 09 18:59:49.261414 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.261388    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-596c6676c9-4cbkx_86bc83f1-769a-4124-bf77-5e919732b28c/main/0.log"
Jul 09 18:59:49.261886 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.261836    2558 generic.go:356] "Generic (PLEG): container finished" podID="86bc83f1-769a-4124-bf77-5e919732b28c" containerID="2dc5a8b4c16a743918178a2e26de0547b66fb38e156e481052a248ea05f25030" exitCode=137
Jul 09 18:59:49.262018 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.261943    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx" event={"ID":"86bc83f1-769a-4124-bf77-5e919732b28c","Type":"ContainerDied","Data":"2dc5a8b4c16a743918178a2e26de0547b66fb38e156e481052a248ea05f25030"}
Jul 09 18:59:49.327075 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.327056    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-596c6676c9-4cbkx_86bc83f1-769a-4124-bf77-5e919732b28c/main/0.log"
Jul 09 18:59:49.328737 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.328714    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:59:49.469740 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.469648    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-home\") pod \"86bc83f1-769a-4124-bf77-5e919732b28c\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") "
Jul 09 18:59:49.469740 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.469682    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-model-cache\") pod \"86bc83f1-769a-4124-bf77-5e919732b28c\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") "
Jul 09 18:59:49.469740 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.469715    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6wpt\" (UniqueName: \"kubernetes.io/projected/86bc83f1-769a-4124-bf77-5e919732b28c-kube-api-access-t6wpt\") pod \"86bc83f1-769a-4124-bf77-5e919732b28c\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") "
Jul 09 18:59:49.470028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.469766    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-tmp-dir\") pod \"86bc83f1-769a-4124-bf77-5e919732b28c\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") "
Jul 09 18:59:49.470028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.469812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-dshm\") pod \"86bc83f1-769a-4124-bf77-5e919732b28c\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") "
Jul 09 18:59:49.470028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.469840    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-kserve-provision-location\") pod \"86bc83f1-769a-4124-bf77-5e919732b28c\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") "
Jul 09 18:59:49.470028 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.469868    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc83f1-769a-4124-bf77-5e919732b28c-tls-certs\") pod \"86bc83f1-769a-4124-bf77-5e919732b28c\" (UID: \"86bc83f1-769a-4124-bf77-5e919732b28c\") "
Jul 09 18:59:49.470243 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.470004    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-model-cache" (OuterVolumeSpecName: "model-cache") pod "86bc83f1-769a-4124-bf77-5e919732b28c" (UID: "86bc83f1-769a-4124-bf77-5e919732b28c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:59:49.470243 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.470177    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-home" (OuterVolumeSpecName: "home") pod "86bc83f1-769a-4124-bf77-5e919732b28c" (UID: "86bc83f1-769a-4124-bf77-5e919732b28c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:59:49.470243 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.470194    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86bc83f1-769a-4124-bf77-5e919732b28c" (UID: "86bc83f1-769a-4124-bf77-5e919732b28c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:59:49.472159 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.472132    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86bc83f1-769a-4124-bf77-5e919732b28c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86bc83f1-769a-4124-bf77-5e919732b28c" (UID: "86bc83f1-769a-4124-bf77-5e919732b28c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:59:49.472159 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.472151    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-dshm" (OuterVolumeSpecName: "dshm") pod "86bc83f1-769a-4124-bf77-5e919732b28c" (UID: "86bc83f1-769a-4124-bf77-5e919732b28c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:59:49.472317 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.472151    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bc83f1-769a-4124-bf77-5e919732b28c-kube-api-access-t6wpt" (OuterVolumeSpecName: "kube-api-access-t6wpt") pod "86bc83f1-769a-4124-bf77-5e919732b28c" (UID: "86bc83f1-769a-4124-bf77-5e919732b28c"). InnerVolumeSpecName "kube-api-access-t6wpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:59:49.528067 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.528016    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86bc83f1-769a-4124-bf77-5e919732b28c" (UID: "86bc83f1-769a-4124-bf77-5e919732b28c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:59:49.570752 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.570718    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:59:49.570752 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.570752    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-kserve-provision-location\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:59:49.570896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.570762    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86bc83f1-769a-4124-bf77-5e919732b28c-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:59:49.570896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.570770    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:59:49.570896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.570778    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:59:49.570896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.570786    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6wpt\" (UniqueName: \"kubernetes.io/projected/86bc83f1-769a-4124-bf77-5e919732b28c-kube-api-access-t6wpt\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:59:49.570896 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:49.570794    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86bc83f1-769a-4124-bf77-5e919732b28c-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 18:59:50.266866 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:50.266839    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-596c6676c9-4cbkx_86bc83f1-769a-4124-bf77-5e919732b28c/main/0.log"
Jul 09 18:59:50.267341 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:50.267253    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"
Jul 09 18:59:50.267341 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:50.267254    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx" event={"ID":"86bc83f1-769a-4124-bf77-5e919732b28c","Type":"ContainerDied","Data":"1fa8623dfcc86a00943664e029aa9393ceb55eac10bffa0c32efdc43227b9bb1"}
Jul 09 18:59:50.267476 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:50.267341    2558 scope.go:117] "RemoveContainer" containerID="2dc5a8b4c16a743918178a2e26de0547b66fb38e156e481052a248ea05f25030"
Jul 09 18:59:50.276043 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:50.276026    2558 scope.go:117] "RemoveContainer" containerID="0162f6c82cb9a4de42c7ab9df470f3c0372a68f46e4875286682d1543abbf079"
Jul 09 18:59:50.290744 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:50.290716    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"]
Jul 09 18:59:50.295102 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:50.295083    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-596c6676c9-4cbkx"]
Jul 09 18:59:50.892430 ip-10-0-140-224 kubenswrapper[2558]: I0709 18:59:50.892399    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86bc83f1-769a-4124-bf77-5e919732b28c" path="/var/lib/kubelet/pods/86bc83f1-769a-4124-bf77-5e919732b28c/volumes"
Jul 09 19:00:19.407797 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.407719    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"]
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.407972    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86bc83f1-769a-4124-bf77-5e919732b28c" containerName="main"
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.407982    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="86bc83f1-769a-4124-bf77-5e919732b28c" containerName="main"
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.407992    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="storage-initializer"
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.407998    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="storage-initializer"
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.408011    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main"
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.408016    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main"
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.408026    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86bc83f1-769a-4124-bf77-5e919732b28c" containerName="storage-initializer"
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.408031    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="86bc83f1-769a-4124-bf77-5e919732b28c" containerName="storage-initializer"
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.408073    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="76056b0b-e73b-47e7-93cc-685da9f680c7" containerName="main"
Jul 09 19:00:19.408240 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.408082    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="86bc83f1-769a-4124-bf77-5e919732b28c" containerName="main"
Jul 09 19:00:19.409899 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.409880    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.414171 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.414133    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 19:00:19.416207 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.415942    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 09 19:00:19.428869 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.428839    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"]
Jul 09 19:00:19.506487 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.506455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.506487 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.506488    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.506690 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.506516    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.506690 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.506573    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.506690 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.506651    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7113004-38af-4f63-a2d3-30b828ee9460-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.506690 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.506677    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnbx\" (UniqueName: \"kubernetes.io/projected/c7113004-38af-4f63-a2d3-30b828ee9460-kube-api-access-nbnbx\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.607369 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.607336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.607369 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.607376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.607639 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.607404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.607639 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.607421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.607639 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.607453    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7113004-38af-4f63-a2d3-30b828ee9460-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.607639 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.607476    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnbx\" (UniqueName: \"kubernetes.io/projected/c7113004-38af-4f63-a2d3-30b828ee9460-kube-api-access-nbnbx\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.607849 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.607661    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.607849 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.607813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.607932 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.607818    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.609770 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.609750    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.609924 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.609905    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7113004-38af-4f63-a2d3-30b828ee9460-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.617269 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.617249    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnbx\" (UniqueName: \"kubernetes.io/projected/c7113004-38af-4f63-a2d3-30b828ee9460-kube-api-access-nbnbx\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.719931 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.719897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:19.854604 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:19.854582    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"]
Jul 09 19:00:19.856651 ip-10-0-140-224 kubenswrapper[2558]: W0709 19:00:19.856623    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7113004_38af_4f63_a2d3_30b828ee9460.slice/crio-95626262424c75005a679f6828d0d5483268efd00d9c6252e84c0aa19fe02d16 WatchSource:0}: Error finding container 95626262424c75005a679f6828d0d5483268efd00d9c6252e84c0aa19fe02d16: Status 404 returned error can't find the container with id 95626262424c75005a679f6828d0d5483268efd00d9c6252e84c0aa19fe02d16
Jul 09 19:00:20.390769 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:20.390737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6" event={"ID":"c7113004-38af-4f63-a2d3-30b828ee9460","Type":"ContainerStarted","Data":"02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c"}
Jul 09 19:00:20.390933 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:20.390788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6" event={"ID":"c7113004-38af-4f63-a2d3-30b828ee9460","Type":"ContainerStarted","Data":"95626262424c75005a679f6828d0d5483268efd00d9c6252e84c0aa19fe02d16"}
Jul 09 19:00:20.418968 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:20.418909    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6" podStartSLOduration=1.418892909 podStartE2EDuration="1.418892909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:20.416401548 +0000 UTC m=+3416.062395092" watchObservedRunningTime="2026-07-09 19:00:20.418892909 +0000 UTC m=+3416.064886485"
Jul 09 19:00:20.720942 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:20.720898    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:29.720687 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:29.720643    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:29.733091 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:29.733063    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:00:30.447195 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:00:30.447166    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:01:11.383590 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.383555    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"]
Jul 09 19:01:11.385664 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.385648    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.389013 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.388996    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 09 19:01:11.398257 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.398232    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"]
Jul 09 19:01:11.521514 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.521481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.521514 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.521515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.521743 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.521544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.521743 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.521648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slqx6\" (UniqueName: \"kubernetes.io/projected/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-kube-api-access-slqx6\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.521743 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.521685    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.521743 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.521720    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.622703 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.622670    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.622896 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.622741    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slqx6\" (UniqueName: \"kubernetes.io/projected/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-kube-api-access-slqx6\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.622896 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.622774    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.622896 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.622813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.623076 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.622978    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.623220 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.623194    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.623623 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.623562    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.623732 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.623581    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.623732 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.623610    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.625654 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.625627    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.625921 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.625903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.632803 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.632780    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slqx6\" (UniqueName: \"kubernetes.io/projected/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-kube-api-access-slqx6\") pod \"rolling-upgrade-test-kserve-859df57bcb-cclrx\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.695918 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.695895    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:11.831999 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:11.831976    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"]
Jul 09 19:01:11.834008 ip-10-0-140-224 kubenswrapper[2558]: W0709 19:01:11.833982    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b4c448_97d1_4a51_bfa1_ec7f414b6ea3.slice/crio-2a9dbc13775d0dff062397f65fe27b78d476e2c230f25136203813be94a846e3 WatchSource:0}: Error finding container 2a9dbc13775d0dff062397f65fe27b78d476e2c230f25136203813be94a846e3: Status 404 returned error can't find the container with id 2a9dbc13775d0dff062397f65fe27b78d476e2c230f25136203813be94a846e3
Jul 09 19:01:12.243695 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:12.243664    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"]
Jul 09 19:01:12.611077 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:12.610987    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx" event={"ID":"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3","Type":"ContainerStarted","Data":"517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f"}
Jul 09 19:01:12.611077 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:12.611047    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx" event={"ID":"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3","Type":"ContainerStarted","Data":"2a9dbc13775d0dff062397f65fe27b78d476e2c230f25136203813be94a846e3"}
Jul 09 19:01:12.634814 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:12.634747    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx" podStartSLOduration=1.634727527 podStartE2EDuration="1.634727527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:12.633482801 +0000 UTC m=+3468.279476352" watchObservedRunningTime="2026-07-09 19:01:12.634727527 +0000 UTC m=+3468.280721068"
Jul 09 19:01:12.696060 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:12.696031    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:12.725223 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:12.725189    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"]
Jul 09 19:01:12.733301 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:12.733272    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:12.733455 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:12.733362    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs podName:d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3 nodeName:}" failed. No retries permitted until 2026-07-09 19:01:13.233339975 +0000 UTC m=+3468.879333505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-cclrx" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:13.237622 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:13.237584    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:13.237811 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:13.237666    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs podName:d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3 nodeName:}" failed. No retries permitted until 2026-07-09 19:01:14.237651408 +0000 UTC m=+3469.883644925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-cclrx" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:14.246308 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:14.246271    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:14.246706 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:14.246362    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs podName:d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3 nodeName:}" failed. No retries permitted until 2026-07-09 19:01:16.246344045 +0000 UTC m=+3471.892337579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-cclrx" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:16.264404 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:16.264366    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:16.264790 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:16.264445    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs podName:d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3 nodeName:}" failed. No retries permitted until 2026-07-09 19:01:20.264430159 +0000 UTC m=+3475.910423676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-cclrx" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:20.298502 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:20.298458    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:20.298994 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:20.298533    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs podName:d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3 nodeName:}" failed. No retries permitted until 2026-07-09 19:01:28.298518891 +0000 UTC m=+3483.944512409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-cclrx" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:27.266245 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.266184    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6" podUID="c7113004-38af-4f63-a2d3-30b828ee9460" containerName="main" containerID="cri-o://02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c" gracePeriod=45
Jul 09 19:01:27.515632 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.515603    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:01:27.551683 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.551604    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-dshm\") pod \"c7113004-38af-4f63-a2d3-30b828ee9460\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") "
Jul 09 19:01:27.551683 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.551647    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7113004-38af-4f63-a2d3-30b828ee9460-tls-certs\") pod \"c7113004-38af-4f63-a2d3-30b828ee9460\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") "
Jul 09 19:01:27.551847 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.551694    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbnbx\" (UniqueName: \"kubernetes.io/projected/c7113004-38af-4f63-a2d3-30b828ee9460-kube-api-access-nbnbx\") pod \"c7113004-38af-4f63-a2d3-30b828ee9460\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") "
Jul 09 19:01:27.551847 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.551741    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-tmp-dir\") pod \"c7113004-38af-4f63-a2d3-30b828ee9460\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") "
Jul 09 19:01:27.551847 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.551786    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-model-cache\") pod \"c7113004-38af-4f63-a2d3-30b828ee9460\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") "
Jul 09 19:01:27.552007 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.551850    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-home\") pod \"c7113004-38af-4f63-a2d3-30b828ee9460\" (UID: \"c7113004-38af-4f63-a2d3-30b828ee9460\") "
Jul 09 19:01:27.552066 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.552033    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7113004-38af-4f63-a2d3-30b828ee9460" (UID: "c7113004-38af-4f63-a2d3-30b828ee9460"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:27.552162 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.552132    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7113004-38af-4f63-a2d3-30b828ee9460" (UID: "c7113004-38af-4f63-a2d3-30b828ee9460"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:27.552247 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.552154    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:27.552247 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.552220    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-home" (OuterVolumeSpecName: "home") pod "c7113004-38af-4f63-a2d3-30b828ee9460" (UID: "c7113004-38af-4f63-a2d3-30b828ee9460"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:27.553899 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.553879    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-dshm" (OuterVolumeSpecName: "dshm") pod "c7113004-38af-4f63-a2d3-30b828ee9460" (UID: "c7113004-38af-4f63-a2d3-30b828ee9460"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:27.553962 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.553921    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7113004-38af-4f63-a2d3-30b828ee9460-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7113004-38af-4f63-a2d3-30b828ee9460" (UID: "c7113004-38af-4f63-a2d3-30b828ee9460"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:01:27.554001 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.553955    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7113004-38af-4f63-a2d3-30b828ee9460-kube-api-access-nbnbx" (OuterVolumeSpecName: "kube-api-access-nbnbx") pod "c7113004-38af-4f63-a2d3-30b828ee9460" (UID: "c7113004-38af-4f63-a2d3-30b828ee9460"). InnerVolumeSpecName "kube-api-access-nbnbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:01:27.653347 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.653289    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:27.653347 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.653315    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:27.653347 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.653349    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7113004-38af-4f63-a2d3-30b828ee9460-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:27.653587 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.653362    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbnbx\" (UniqueName: \"kubernetes.io/projected/c7113004-38af-4f63-a2d3-30b828ee9460-kube-api-access-nbnbx\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:27.653587 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.653370    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7113004-38af-4f63-a2d3-30b828ee9460-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:27.671910 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.671878    2558 generic.go:356] "Generic (PLEG): container finished" podID="c7113004-38af-4f63-a2d3-30b828ee9460" containerID="02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c" exitCode=0
Jul 09 19:01:27.672062 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.671927    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"
Jul 09 19:01:27.672062 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.671962    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6" event={"ID":"c7113004-38af-4f63-a2d3-30b828ee9460","Type":"ContainerDied","Data":"02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c"}
Jul 09 19:01:27.672062 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.672018    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6" event={"ID":"c7113004-38af-4f63-a2d3-30b828ee9460","Type":"ContainerDied","Data":"95626262424c75005a679f6828d0d5483268efd00d9c6252e84c0aa19fe02d16"}
Jul 09 19:01:27.672062 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.672039    2558 scope.go:117] "RemoveContainer" containerID="02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c"
Jul 09 19:01:27.680744 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.680725    2558 scope.go:117] "RemoveContainer" containerID="02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c"
Jul 09 19:01:27.680987 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:27.680969    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c\": container with ID starting with 02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c not found: ID does not exist" containerID="02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c"
Jul 09 19:01:27.681049 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.680999    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c"} err="failed to get container status \"02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c\": rpc error: code = NotFound desc = could not find container \"02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c\": container with ID starting with 02f60b45fd0ae465a689c30d39f8f8abef03816ad9375512cd144af0151df03c not found: ID does not exist"
Jul 09 19:01:27.700834 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.700809    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"]
Jul 09 19:01:27.704398 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:27.704378    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6"]
Jul 09 19:01:28.358366 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:28.358312    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:28.358835 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:28.358401    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs podName:d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3 nodeName:}" failed. No retries permitted until 2026-07-09 19:01:44.358387398 +0000 UTC m=+3500.004380914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-cclrx" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 09 19:01:28.437477 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:28.437445    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-cfdk6" podUID="c7113004-38af-4f63-a2d3-30b828ee9460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 09 19:01:28.892483 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:28.892453    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7113004-38af-4f63-a2d3-30b828ee9460" path="/var/lib/kubelet/pods/c7113004-38af-4f63-a2d3-30b828ee9460/volumes"
Jul 09 19:01:29.639116 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.639079    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx" podUID="d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" containerName="main" containerID="cri-o://517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f" gracePeriod=45
Jul 09 19:01:29.881452 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.881425    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:29.972641 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.972613    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-dshm\") pod \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") "
Jul 09 19:01:29.972803 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.972660    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slqx6\" (UniqueName: \"kubernetes.io/projected/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-kube-api-access-slqx6\") pod \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") "
Jul 09 19:01:29.972803 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.972681    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-home\") pod \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") "
Jul 09 19:01:29.972803 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.972722    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs\") pod \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") "
Jul 09 19:01:29.972803 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.972746    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tmp-dir\") pod \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") "
Jul 09 19:01:29.972803 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.972776    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-model-cache\") pod \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\" (UID: \"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3\") "
Jul 09 19:01:29.973048 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.972993    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:29.973048 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.973006    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-home" (OuterVolumeSpecName: "home") pod "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:29.973173 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.973110    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:29.974963 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.974941    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:01:29.974963 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.974945    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-dshm" (OuterVolumeSpecName: "dshm") pod "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:29.974963 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:29.974958    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-kube-api-access-slqx6" (OuterVolumeSpecName: "kube-api-access-slqx6") pod "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" (UID: "d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3"). InnerVolumeSpecName "kube-api-access-slqx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:01:30.073573 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.073535    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:30.073573 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.073568    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:30.073573 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.073578    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:30.073573 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.073586    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:30.073801 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.073595    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slqx6\" (UniqueName: \"kubernetes.io/projected/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-kube-api-access-slqx6\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:30.073801 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.073602    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:01:30.684963 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.684923    2558 generic.go:356] "Generic (PLEG): container finished" podID="d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" containerID="517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f" exitCode=0
Jul 09 19:01:30.684963 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.684966    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"
Jul 09 19:01:30.685458 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.685007    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx" event={"ID":"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3","Type":"ContainerDied","Data":"517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f"}
Jul 09 19:01:30.685458 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.685066    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx" event={"ID":"d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3","Type":"ContainerDied","Data":"2a9dbc13775d0dff062397f65fe27b78d476e2c230f25136203813be94a846e3"}
Jul 09 19:01:30.685458 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.685088    2558 scope.go:117] "RemoveContainer" containerID="517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f"
Jul 09 19:01:30.693507 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.693465    2558 scope.go:117] "RemoveContainer" containerID="517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f"
Jul 09 19:01:30.693746 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:01:30.693722    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f\": container with ID starting with 517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f not found: ID does not exist" containerID="517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f"
Jul 09 19:01:30.693859 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.693752    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f"} err="failed to get container status \"517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f\": rpc error: code = NotFound desc = could not find container \"517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f\": container with ID starting with 517218bb8e573353f1a726e18e366eca473114ccb44bf7cc1aac8c66eb23732f not found: ID does not exist"
Jul 09 19:01:30.719472 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.719435    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"]
Jul 09 19:01:30.727171 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.727142    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-cclrx"]
Jul 09 19:01:30.893006 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:30.892978    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" path="/var/lib/kubelet/pods/d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3/volumes"
Jul 09 19:01:41.474618 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.474586    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"]
Jul 09 19:01:41.475007 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.474841    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7113004-38af-4f63-a2d3-30b828ee9460" containerName="main"
Jul 09 19:01:41.475007 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.474851    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7113004-38af-4f63-a2d3-30b828ee9460" containerName="main"
Jul 09 19:01:41.475007 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.474869    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" containerName="main"
Jul 09 19:01:41.475007 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.474874    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" containerName="main"
Jul 09 19:01:41.475007 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.474916    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7113004-38af-4f63-a2d3-30b828ee9460" containerName="main"
Jul 09 19:01:41.475007 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.474923    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7b4c448-97d1-4a51-bfa1-ec7f414b6ea3" containerName="main"
Jul 09 19:01:41.477272 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.477255    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.480848 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.480826    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 19:01:41.482437 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.482417    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-epp-sa-dockercfg-fp8rv\""
Jul 09 19:01:41.482564 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.482467    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 09 19:01:41.488771 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.488748    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"]
Jul 09 19:01:41.557401 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.557361    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.557607 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.557404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwvd\" (UniqueName: \"kubernetes.io/projected/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-kube-api-access-wcwvd\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.557607 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.557460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.557607 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.557495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.557607 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.557537    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.658727 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.658688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.658909 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.658740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.658909 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.658767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwvd\" (UniqueName: \"kubernetes.io/projected/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-kube-api-access-wcwvd\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.658909 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.658795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.658909 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.658814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.659110 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.659091    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.659174 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.659153    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.659263 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.659246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.661411 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.661392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.669161 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.669134    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwvd\" (UniqueName: \"kubernetes.io/projected/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-kube-api-access-wcwvd\") pod \"scheduler-custom-template-test-kserve-router-scheduler-56797bz4\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.786023 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.785929    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:41.929306 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.929278    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"]
Jul 09 19:01:41.930148 ip-10-0-140-224 kubenswrapper[2558]: W0709 19:01:41.930124    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a67f62_35d2_41b5_8bd6_bb32dc9a046e.slice/crio-1929489ee25e4840e84bd65da08caad646691a93749a734262acd5b025307c8f WatchSource:0}: Error finding container 1929489ee25e4840e84bd65da08caad646691a93749a734262acd5b025307c8f: Status 404 returned error can't find the container with id 1929489ee25e4840e84bd65da08caad646691a93749a734262acd5b025307c8f
Jul 09 19:01:41.932710 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:41.932688    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:01:42.739713 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:42.739669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4" event={"ID":"23a67f62-35d2-41b5-8bd6-bb32dc9a046e","Type":"ContainerStarted","Data":"1929489ee25e4840e84bd65da08caad646691a93749a734262acd5b025307c8f"}
Jul 09 19:01:43.745787 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:43.745741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4" event={"ID":"23a67f62-35d2-41b5-8bd6-bb32dc9a046e","Type":"ContainerStarted","Data":"de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00"}
Jul 09 19:01:43.746219 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:43.745812    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:01:43.767434 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:01:43.767385    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4" podStartSLOduration=1.676067789 podStartE2EDuration="2.767370553s" totalImagesPullingTime="1.091302764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:43.766389876 +0000 UTC m=+3499.412383417" watchObservedRunningTime="2026-07-09 19:01:43.767370553 +0000 UTC m=+3499.413364089"
Jul 09 19:02:14.751733 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:14.751655    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:02:29.817836 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:29.817798    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"]
Jul 09 19:02:29.826775 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:02:29.826742    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 19:02:29.826775 ip-10-0-140-224 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:02:29.826775 ip-10-0-140-224 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Jul 09 19:02:29.826775 ip-10-0-140-224 kubenswrapper[2558]:  > containerID="de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00" cmd=["/bin/sleep","15"]
Jul 09 19:02:29.827057 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:29.826791    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4" message="" err=<
Jul 09 19:02:29.827057 ip-10-0-140-224 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:02:29.827057 ip-10-0-140-224 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Jul 09 19:02:29.827057 ip-10-0-140-224 kubenswrapper[2558]:  >
Jul 09 19:02:29.827057 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:02:29.826805    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 19:02:29.827057 ip-10-0-140-224 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 19:02:29.827057 ip-10-0-140-224 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Jul 09 19:02:29.827057 ip-10-0-140-224 kubenswrapper[2558]:  > pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4" podUID="23a67f62-35d2-41b5-8bd6-bb32dc9a046e" containerName="main" containerID="cri-o://de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00"
Jul 09 19:02:29.827057 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:29.826878    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4" podUID="23a67f62-35d2-41b5-8bd6-bb32dc9a046e" containerName="main" containerID="cri-o://de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00" gracePeriod=60
Jul 09 19:02:30.076246 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.076183    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:02:30.162384 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.162357    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-cachi2-cache\") pod \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") "
Jul 09 19:02:30.162384 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.162391    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tls-certs\") pod \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") "
Jul 09 19:02:30.162615 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.162445    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-uds\") pod \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") "
Jul 09 19:02:30.162615 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.162485    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcwvd\" (UniqueName: \"kubernetes.io/projected/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-kube-api-access-wcwvd\") pod \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") "
Jul 09 19:02:30.162615 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.162521    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-cache\") pod \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\" (UID: \"23a67f62-35d2-41b5-8bd6-bb32dc9a046e\") "
Jul 09 19:02:30.162738 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.162658    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "23a67f62-35d2-41b5-8bd6-bb32dc9a046e" (UID: "23a67f62-35d2-41b5-8bd6-bb32dc9a046e"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:02:30.162775 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.162739    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "23a67f62-35d2-41b5-8bd6-bb32dc9a046e" (UID: "23a67f62-35d2-41b5-8bd6-bb32dc9a046e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:02:30.162833 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.162808    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "23a67f62-35d2-41b5-8bd6-bb32dc9a046e" (UID: "23a67f62-35d2-41b5-8bd6-bb32dc9a046e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:02:30.164699 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.164674    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-kube-api-access-wcwvd" (OuterVolumeSpecName: "kube-api-access-wcwvd") pod "23a67f62-35d2-41b5-8bd6-bb32dc9a046e" (UID: "23a67f62-35d2-41b5-8bd6-bb32dc9a046e"). InnerVolumeSpecName "kube-api-access-wcwvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:02:30.164814 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.164736    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23a67f62-35d2-41b5-8bd6-bb32dc9a046e" (UID: "23a67f62-35d2-41b5-8bd6-bb32dc9a046e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:02:30.263641 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.263610    2558 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-uds\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:02:30.263641 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.263636    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcwvd\" (UniqueName: \"kubernetes.io/projected/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-kube-api-access-wcwvd\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:02:30.263641 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.263645    2558 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tokenizer-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:02:30.263872 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.263654    2558 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-cachi2-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:02:30.263872 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.263664    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a67f62-35d2-41b5-8bd6-bb32dc9a046e-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:02:30.935890 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.935859    2558 generic.go:356] "Generic (PLEG): container finished" podID="23a67f62-35d2-41b5-8bd6-bb32dc9a046e" containerID="de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00" exitCode=0
Jul 09 19:02:30.936265 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.935908    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"
Jul 09 19:02:30.936265 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.935941    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4" event={"ID":"23a67f62-35d2-41b5-8bd6-bb32dc9a046e","Type":"ContainerDied","Data":"de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00"}
Jul 09 19:02:30.936265 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.935991    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4" event={"ID":"23a67f62-35d2-41b5-8bd6-bb32dc9a046e","Type":"ContainerDied","Data":"1929489ee25e4840e84bd65da08caad646691a93749a734262acd5b025307c8f"}
Jul 09 19:02:30.936265 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.936010    2558 scope.go:117] "RemoveContainer" containerID="de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00"
Jul 09 19:02:30.944553 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.944538    2558 scope.go:117] "RemoveContainer" containerID="de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00"
Jul 09 19:02:30.944832 ip-10-0-140-224 kubenswrapper[2558]: E0709 19:02:30.944807    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00\": container with ID starting with de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00 not found: ID does not exist" containerID="de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00"
Jul 09 19:02:30.944915 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.944839    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00"} err="failed to get container status \"de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00\": rpc error: code = NotFound desc = could not find container \"de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00\": container with ID starting with de7ec9aeeaebeb57596cb45c167f2a01329afc7af23b627be648084f238acc00 not found: ID does not exist"
Jul 09 19:02:30.956373 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.956348    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"]
Jul 09 19:02:30.961245 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:30.961225    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-56797bz4"]
Jul 09 19:02:32.893847 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:32.893813    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23a67f62-35d2-41b5-8bd6-bb32dc9a046e" path="/var/lib/kubelet/pods/23a67f62-35d2-41b5-8bd6-bb32dc9a046e/volumes"
Jul 09 19:02:46.141186 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.141156    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"]
Jul 09 19:02:46.141682 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.141454    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a67f62-35d2-41b5-8bd6-bb32dc9a046e" containerName="main"
Jul 09 19:02:46.141682 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.141468    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a67f62-35d2-41b5-8bd6-bb32dc9a046e" containerName="main"
Jul 09 19:02:46.141682 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.141542    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a67f62-35d2-41b5-8bd6-bb32dc9a046e" containerName="main"
Jul 09 19:02:46.143536 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.143521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.148617 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.148595    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w96lf\""
Jul 09 19:02:46.148752 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.148599    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 09 19:02:46.155133 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.155109    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"]
Jul 09 19:02:46.287733 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.287703    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.287931 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.287740    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e221823-6723-4fb6-9434-bdb840215d08-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.287931 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.287776    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.287931 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.287835    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.287931 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.287899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2b26\" (UniqueName: \"kubernetes.io/projected/2e221823-6723-4fb6-9434-bdb840215d08-kube-api-access-x2b26\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.288109 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.287962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.388788 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.388753    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b26\" (UniqueName: \"kubernetes.io/projected/2e221823-6723-4fb6-9434-bdb840215d08-kube-api-access-x2b26\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.388967 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.388801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.388967 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.388837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.388967 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.388867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e221823-6723-4fb6-9434-bdb840215d08-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.388967 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.388915    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.389186 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.389056    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.389313 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.389282    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.389459 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.389319    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.389570 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.389457    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.391387 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.391316    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.391603 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.391585    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e221823-6723-4fb6-9434-bdb840215d08-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.399082 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.399063    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2b26\" (UniqueName: \"kubernetes.io/projected/2e221823-6723-4fb6-9434-bdb840215d08-kube-api-access-x2b26\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.452972 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.452943    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:46.591747 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:46.591722    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"]
Jul 09 19:02:46.593996 ip-10-0-140-224 kubenswrapper[2558]: W0709 19:02:46.593970    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e221823_6723_4fb6_9434_bdb840215d08.slice/crio-057b96735495d710e8d7b5eb60aa3c8be463af4c4ca7701dac93255ecae7e5de WatchSource:0}: Error finding container 057b96735495d710e8d7b5eb60aa3c8be463af4c4ca7701dac93255ecae7e5de: Status 404 returned error can't find the container with id 057b96735495d710e8d7b5eb60aa3c8be463af4c4ca7701dac93255ecae7e5de
Jul 09 19:02:47.002101 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:47.002068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw" event={"ID":"2e221823-6723-4fb6-9434-bdb840215d08","Type":"ContainerStarted","Data":"271737e5f95d4ad9f657f79d3c519009c9249d2e9e626ec273b562c74c004efd"}
Jul 09 19:02:47.002282 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:47.002115    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw" event={"ID":"2e221823-6723-4fb6-9434-bdb840215d08","Type":"ContainerStarted","Data":"057b96735495d710e8d7b5eb60aa3c8be463af4c4ca7701dac93255ecae7e5de"}
Jul 09 19:02:47.032608 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:47.032554    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw" podStartSLOduration=1.032540379 podStartE2EDuration="1.032540379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:47.030968712 +0000 UTC m=+3562.676962251" watchObservedRunningTime="2026-07-09 19:02:47.032540379 +0000 UTC m=+3562.678533921"
Jul 09 19:02:47.453609 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:47.453574    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:56.453776 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:56.453736    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:56.465906 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:56.465882    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:02:57.054826 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:02:57.054786    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:03:12.322109 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.322075    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"]
Jul 09 19:03:12.324892 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.324866    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.328638 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.328608    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 09 19:03:12.330165 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.330145    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-zxqsg\""
Jul 09 19:03:12.340338 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.340301    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"]
Jul 09 19:03:12.509007 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.508977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.509007 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.509011    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.509244 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.509087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.509244 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.509121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w887b\" (UniqueName: \"kubernetes.io/projected/8b2f8683-e079-4829-bfbc-75e359df8dac-kube-api-access-w887b\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.509244 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.509156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2f8683-e079-4829-bfbc-75e359df8dac-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.509244 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.509200    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.609838 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.609742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.609838 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.609775    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.609838 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.609806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.609838 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.609824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w887b\" (UniqueName: \"kubernetes.io/projected/8b2f8683-e079-4829-bfbc-75e359df8dac-kube-api-access-w887b\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.609838 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.609841    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2f8683-e079-4829-bfbc-75e359df8dac-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.610215 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.610026    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.610307 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.610277    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.610413 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.610301    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.610413 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.610356    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.612342 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.612310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2f8683-e079-4829-bfbc-75e359df8dac-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.612574 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.612555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2f8683-e079-4829-bfbc-75e359df8dac-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.620274 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.620244    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w887b\" (UniqueName: \"kubernetes.io/projected/8b2f8683-e079-4829-bfbc-75e359df8dac-kube-api-access-w887b\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4\" (UID: \"8b2f8683-e079-4829-bfbc-75e359df8dac\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.635150 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.635124    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:12.785886 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:12.785859    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"]
Jul 09 19:03:12.787717 ip-10-0-140-224 kubenswrapper[2558]: W0709 19:03:12.787684    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2f8683_e079_4829_bfbc_75e359df8dac.slice/crio-9eeedc787b705b0ec8420b233712ed4e8c6a5fb2f3e6e93187e01d63583256de WatchSource:0}: Error finding container 9eeedc787b705b0ec8420b233712ed4e8c6a5fb2f3e6e93187e01d63583256de: Status 404 returned error can't find the container with id 9eeedc787b705b0ec8420b233712ed4e8c6a5fb2f3e6e93187e01d63583256de
Jul 09 19:03:13.107585 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:13.107550    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4" event={"ID":"8b2f8683-e079-4829-bfbc-75e359df8dac","Type":"ContainerStarted","Data":"afe57b550ddc9874b235ee0fa4aaaf89d83b7785a8d4a2390dafb2f9aa0d9cbc"}
Jul 09 19:03:13.107779 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:13.107605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4" event={"ID":"8b2f8683-e079-4829-bfbc-75e359df8dac","Type":"ContainerStarted","Data":"9eeedc787b705b0ec8420b233712ed4e8c6a5fb2f3e6e93187e01d63583256de"}
Jul 09 19:03:13.107779 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:13.107742    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:14.113598 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:14.113555    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4" event={"ID":"8b2f8683-e079-4829-bfbc-75e359df8dac","Type":"ContainerStarted","Data":"f00d0fb149069edac103556c5c8d0fd9a37670dbf786cdb4f0e3ed770f5c5c73"}
Jul 09 19:03:14.137530 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:14.137472    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4" podStartSLOduration=2.137455246 podStartE2EDuration="2.137455246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:14.135952884 +0000 UTC m=+3589.781946423" watchObservedRunningTime="2026-07-09 19:03:14.137455246 +0000 UTC m=+3589.783448785"
Jul 09 19:03:14.635285 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:14.635247    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:22.636172 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:22.636084    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:22.648766 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:22.648742    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:22.650219 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:22.650198    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:23.163005 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:23.162980    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4"
Jul 09 19:03:25.052820 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:25.052786    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:03:25.062356 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:25.062320    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:03:34.095397 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:34.095358    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"]
Jul 09 19:03:49.120961 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.120897    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw" podUID="2e221823-6723-4fb6-9434-bdb840215d08" containerName="main" containerID="cri-o://271737e5f95d4ad9f657f79d3c519009c9249d2e9e626ec273b562c74c004efd" gracePeriod=45
Jul 09 19:03:49.261043 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.261007    2558 generic.go:356] "Generic (PLEG): container finished" podID="2e221823-6723-4fb6-9434-bdb840215d08" containerID="271737e5f95d4ad9f657f79d3c519009c9249d2e9e626ec273b562c74c004efd" exitCode=0
Jul 09 19:03:49.261043 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.261029    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw" event={"ID":"2e221823-6723-4fb6-9434-bdb840215d08","Type":"ContainerDied","Data":"271737e5f95d4ad9f657f79d3c519009c9249d2e9e626ec273b562c74c004efd"}
Jul 09 19:03:49.363466 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.363442    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:03:49.511527 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.511496    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2b26\" (UniqueName: \"kubernetes.io/projected/2e221823-6723-4fb6-9434-bdb840215d08-kube-api-access-x2b26\") pod \"2e221823-6723-4fb6-9434-bdb840215d08\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") "
Jul 09 19:03:49.511703 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.511568    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e221823-6723-4fb6-9434-bdb840215d08-tls-certs\") pod \"2e221823-6723-4fb6-9434-bdb840215d08\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") "
Jul 09 19:03:49.511749 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.511728    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-home\") pod \"2e221823-6723-4fb6-9434-bdb840215d08\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") "
Jul 09 19:03:49.511789 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.511767    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-dshm\") pod \"2e221823-6723-4fb6-9434-bdb840215d08\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") "
Jul 09 19:03:49.511789 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.511783    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-model-cache\") pod \"2e221823-6723-4fb6-9434-bdb840215d08\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") "
Jul 09 19:03:49.511885 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.511806    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-tmp-dir\") pod \"2e221823-6723-4fb6-9434-bdb840215d08\" (UID: \"2e221823-6723-4fb6-9434-bdb840215d08\") "
Jul 09 19:03:49.511995 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.511970    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-home" (OuterVolumeSpecName: "home") pod "2e221823-6723-4fb6-9434-bdb840215d08" (UID: "2e221823-6723-4fb6-9434-bdb840215d08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:49.512148 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.512060    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-home\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:03:49.512148 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.512063    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e221823-6723-4fb6-9434-bdb840215d08" (UID: "2e221823-6723-4fb6-9434-bdb840215d08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:49.512148 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.512129    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e221823-6723-4fb6-9434-bdb840215d08" (UID: "2e221823-6723-4fb6-9434-bdb840215d08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:49.514036 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.514010    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e221823-6723-4fb6-9434-bdb840215d08-kube-api-access-x2b26" (OuterVolumeSpecName: "kube-api-access-x2b26") pod "2e221823-6723-4fb6-9434-bdb840215d08" (UID: "2e221823-6723-4fb6-9434-bdb840215d08"). InnerVolumeSpecName "kube-api-access-x2b26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:03:49.514198 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.514167    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e221823-6723-4fb6-9434-bdb840215d08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e221823-6723-4fb6-9434-bdb840215d08" (UID: "2e221823-6723-4fb6-9434-bdb840215d08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:03:49.514299 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.514222    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-dshm" (OuterVolumeSpecName: "dshm") pod "2e221823-6723-4fb6-9434-bdb840215d08" (UID: "2e221823-6723-4fb6-9434-bdb840215d08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:03:49.612876 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.612843    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e221823-6723-4fb6-9434-bdb840215d08-tls-certs\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:03:49.612876 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.612874    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-dshm\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:03:49.612876 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.612886    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-model-cache\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:03:49.613100 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.612894    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e221823-6723-4fb6-9434-bdb840215d08-tmp-dir\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:03:49.613100 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:49.612905    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2b26\" (UniqueName: \"kubernetes.io/projected/2e221823-6723-4fb6-9434-bdb840215d08-kube-api-access-x2b26\") on node \"ip-10-0-140-224.ec2.internal\" DevicePath \"\""
Jul 09 19:03:50.265971 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:50.265934    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw" event={"ID":"2e221823-6723-4fb6-9434-bdb840215d08","Type":"ContainerDied","Data":"057b96735495d710e8d7b5eb60aa3c8be463af4c4ca7701dac93255ecae7e5de"}
Jul 09 19:03:50.265971 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:50.265976    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"
Jul 09 19:03:50.266490 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:50.266026    2558 scope.go:117] "RemoveContainer" containerID="271737e5f95d4ad9f657f79d3c519009c9249d2e9e626ec273b562c74c004efd"
Jul 09 19:03:50.306672 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:50.306643    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"]
Jul 09 19:03:50.320771 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:50.320743    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dp5cw"]
Jul 09 19:03:50.892647 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:03:50.892617    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e221823-6723-4fb6-9434-bdb840215d08" path="/var/lib/kubelet/pods/2e221823-6723-4fb6-9434-bdb840215d08/volumes"
Jul 09 19:08:25.075834 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:08:25.075752    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:08:25.085337 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:08:25.085302    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:13:25.098835 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:13:25.098721    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:13:25.109703 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:13:25.109682    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:18:25.120902 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:18:25.120795    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:18:25.132756 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:18:25.132732    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:22:10.589891 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:10.589836    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:10.599650 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:10.599627    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:30.459563 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:30.459525    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:30.460231 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:30.460212    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:30.496400 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:30.496376    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:30.535776 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:30.535752    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:32.101212 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:32.101176    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:32.102146 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:32.102125    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:32.132550 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:32.132524    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:32.147946 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:32.147922    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:33.725995 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:33.725964    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:33.726879 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:33.726857    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:33.752800 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:33.752777    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:33.768375 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:33.768348    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:35.331730 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:35.331702    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:35.332500 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:35.332474    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:35.363697 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:35.363671    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:35.381044 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:35.381021    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:36.942224 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:36.942187    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:36.942985 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:36.942959    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:36.972530 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:36.972499    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:36.989470 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:36.989434    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:38.585519 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:38.585472    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:38.586303 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:38.586284    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:38.621043 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:38.621010    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:38.637285 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:38.637250    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:40.187021 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:40.186993    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:40.187965 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:40.187930    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:40.216754 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:40.216728    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:40.236829 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:40.236805    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:41.922054 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:41.922023    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:41.922749 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:41.922733    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:41.962309 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:41.962282    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:41.979543 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:41.979488    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:43.662112 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:43.662089    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:43.662972 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:43.662906    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:43.695457 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:43.695434    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:43.713955 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:43.713927    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:45.268657 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:45.268617    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:45.269369 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:45.269346    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:45.316736 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:45.316707    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:45.331822 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:45.331788    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:46.920649 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:46.920615    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:46.921284 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:46.921262    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:46.948159 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:46.948131    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:46.961982 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:46.961935    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:48.599910 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:48.599878    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:48.600645 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:48.600628    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:48.634444 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:48.634422    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:48.659283 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:48.659261    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:50.326958 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:50.326928    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:50.327817 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:50.327778    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:50.383610 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:50.383584    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:50.402760 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:50.402736    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:51.960400 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:51.960372    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:51.961193 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:51.961172    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:51.988476 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:51.988451    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:52.003401 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:52.003383    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:53.599698 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:53.599670    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:53.600580 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:53.600550    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:53.629315 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:53.629290    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:53.647168 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:53.647139    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:22:55.264572 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:55.264539    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/0.log"
Jul 09 19:22:55.265396 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:55.265372    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-755f5_35bb2f54-cb55-4392-8f28-8b2709b69211/istio-proxy/1.log"
Jul 09 19:22:55.334299 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:55.334270    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/main/0.log"
Jul 09 19:22:55.362879 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:22:55.362851    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-dwjs4_8b2f8683-e079-4829-bfbc-75e359df8dac/llm-d-routing-sidecar/0.log"
Jul 09 19:23:02.085594 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:02.085558    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:02.333087 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:02.333054    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:03.454312 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:03.454278    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:03.681468 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:03.681438    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:04.792979 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:04.792943    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:05.036090 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:05.036052    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:06.102012 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:06.101987    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:06.356970 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:06.356888    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:07.458024 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:07.457993    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:07.697015 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:07.696982    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:08.798285 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:08.798255    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:09.050642 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:09.050559    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:10.141192 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:10.141160    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:10.386465 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:10.386431    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:11.503299 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:11.503267    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:11.769656 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:11.769568    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:12.986580 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:12.986546    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:13.233206 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:13.233172    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:14.328900 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:14.328867    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:14.578999 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:14.578906    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:15.659483 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:15.659448    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:15.893809 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:15.893778    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:21.633979 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:21.633954    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ql4td_283ad84b-60d3-43ab-80e6-33da3e63bc77/global-pull-secret-syncer/0.log"
Jul 09 19:23:21.777245 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:21.777207    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dgwzk_a35e3606-7da3-4b0a-8c77-ebcef77fc51e/konnectivity-agent/0.log"
Jul 09 19:23:21.800395 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:21.800371    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-224.ec2.internal_d2fa477e951a5bf596a004d58a807d25/haproxy/0.log"
Jul 09 19:23:25.143952 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:25.143841    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:23:25.157545 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:25.157523    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5s8lz_6b892b08-6ba5-42bc-84b6-85f34f7d8119/ovn-acl-logging/0.log"
Jul 09 19:23:26.468267 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:26.468231    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-8rb4q_ae920be6-20a5-4809-971f-de2a1417a44b/authorino/0.log"
Jul 09 19:23:26.778701 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:26.778619    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-4x4rk_f848c811-6e04-4155-a455-063e0d63d7d5/manager/0.log"
Jul 09 19:23:28.776491 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:28.776463    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bjwn_8e9235fc-c082-4977-b7e9-a8d08b751910/node-exporter/0.log"
Jul 09 19:23:28.802929 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:28.802906    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bjwn_8e9235fc-c082-4977-b7e9-a8d08b751910/kube-rbac-proxy/0.log"
Jul 09 19:23:28.843048 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:28.843017    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5bjwn_8e9235fc-c082-4977-b7e9-a8d08b751910/init-textfile/0.log"
Jul 09 19:23:31.352147 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.352116    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"]
Jul 09 19:23:31.352584 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.352390    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e221823-6723-4fb6-9434-bdb840215d08" containerName="main"
Jul 09 19:23:31.352584 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.352401    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e221823-6723-4fb6-9434-bdb840215d08" containerName="main"
Jul 09 19:23:31.352584 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.352459    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e221823-6723-4fb6-9434-bdb840215d08" containerName="main"
Jul 09 19:23:31.355046 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.355029    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.358259 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.358240    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vv9mp\"/\"kube-root-ca.crt\""
Jul 09 19:23:31.359888 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.359868    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vv9mp\"/\"openshift-service-ca.crt\""
Jul 09 19:23:31.359973 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.359869    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vv9mp\"/\"default-dockercfg-rrxh9\""
Jul 09 19:23:31.366504 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.366481    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"]
Jul 09 19:23:31.537348 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.537284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-lib-modules\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.537541 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.537366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-proc\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.537541 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.537405    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-podres\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.537541 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.537426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dql2q\" (UniqueName: \"kubernetes.io/projected/8e96bec4-477d-430c-90cc-b1d3a85a1b18-kube-api-access-dql2q\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.537541 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.537444    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-sys\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.638294 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.638199    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-sys\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.638294 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.638250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-lib-modules\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.638294 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.638293    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-proc\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.638549 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.638316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-podres\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.638549 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.638360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dql2q\" (UniqueName: \"kubernetes.io/projected/8e96bec4-477d-430c-90cc-b1d3a85a1b18-kube-api-access-dql2q\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.638549 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.638359    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-sys\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.638549 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.638429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-proc\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.638549 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.638443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-lib-modules\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.638549 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.638482    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e96bec4-477d-430c-90cc-b1d3a85a1b18-podres\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.648530 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.648506    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dql2q\" (UniqueName: \"kubernetes.io/projected/8e96bec4-477d-430c-90cc-b1d3a85a1b18-kube-api-access-dql2q\") pod \"perf-node-gather-daemonset-65zdj\" (UID: \"8e96bec4-477d-430c-90cc-b1d3a85a1b18\") " pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.665466 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.665443    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:31.795669 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:31.795644    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"]
Jul 09 19:23:31.798459 ip-10-0-140-224 kubenswrapper[2558]: W0709 19:23:31.798430    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8e96bec4_477d_430c_90cc_b1d3a85a1b18.slice/crio-4bec2143cc96c3821b66fce1acf6560c9c98e20fefccea7540715f59f5f2b57f WatchSource:0}: Error finding container 4bec2143cc96c3821b66fce1acf6560c9c98e20fefccea7540715f59f5f2b57f: Status 404 returned error can't find the container with id 4bec2143cc96c3821b66fce1acf6560c9c98e20fefccea7540715f59f5f2b57f
Jul 09 19:23:32.083028 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:32.082997    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj" event={"ID":"8e96bec4-477d-430c-90cc-b1d3a85a1b18","Type":"ContainerStarted","Data":"b26bd15d7f4afbb92e23bab652f1441d111052629ab2d93cf5e32cf7e84068dd"}
Jul 09 19:23:32.083206 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:32.083071    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj" event={"ID":"8e96bec4-477d-430c-90cc-b1d3a85a1b18","Type":"ContainerStarted","Data":"4bec2143cc96c3821b66fce1acf6560c9c98e20fefccea7540715f59f5f2b57f"}
Jul 09 19:23:32.083206 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:32.083180    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:32.112224 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:32.112179    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj" podStartSLOduration=1.112164662 podStartE2EDuration="1.112164662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:23:32.111359409 +0000 UTC m=+4807.757352941" watchObservedRunningTime="2026-07-09 19:23:32.112164662 +0000 UTC m=+4807.758158201"
Jul 09 19:23:33.451629 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:33.451603    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b2gwz_ec64976d-0e4f-4c30-9203-b000c1597408/dns/0.log"
Jul 09 19:23:33.479900 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:33.479873    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b2gwz_ec64976d-0e4f-4c30-9203-b000c1597408/kube-rbac-proxy/0.log"
Jul 09 19:23:33.665808 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:33.665777    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qpt8z_968af6c7-fb8f-4937-a490-2f31ecdae078/dns-node-resolver/0.log"
Jul 09 19:23:34.237414 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:34.237379    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7775456cb8-9znrq_97da9de6-c59a-452f-b85e-8d92de447407/registry/0.log"
Jul 09 19:23:34.270800 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:34.270774    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ft4xk_cd1e2564-98b3-4035-819c-830ebcc71c13/node-ca/0.log"
Jul 09 19:23:36.277781 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:36.277750    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9gflx_f446a6a5-d53a-4147-a454-18dc8917eadd/serve-healthcheck-canary/0.log"
Jul 09 19:23:36.866934 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:36.866903    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmvpf_b472c8fe-f511-4965-91fa-237ad1fd8f6d/kube-rbac-proxy/0.log"
Jul 09 19:23:36.889186 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:36.889158    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmvpf_b472c8fe-f511-4965-91fa-237ad1fd8f6d/exporter/0.log"
Jul 09 19:23:36.916706 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:36.916673    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmvpf_b472c8fe-f511-4965-91fa-237ad1fd8f6d/extractor/0.log"
Jul 09 19:23:38.097871 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:38.097845    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vv9mp/perf-node-gather-daemonset-65zdj"
Jul 09 19:23:40.125723 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:40.125698    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7885457d66-lwd5g_68ca30f0-e90d-4f03-88fb-d68842722870/manager/0.log"
Jul 09 19:23:40.167148 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:40.167115    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-rdqn6_5563bfaf-42e8-4827-b6af-c9571c348468/openshift-lws-operator/0.log"
Jul 09 19:23:41.718867 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:41.718834    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hp242_0805d820-0233-4118-bc76-20e1c70e108c/s3-init/0.log"
Jul 09 19:23:41.740379 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:41.740354    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hp242_0805d820-0233-4118-bc76-20e1c70e108c/download-hf-model/0.log"
Jul 09 19:23:48.323285 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:48.323242    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-54sxh_b9c81c1a-2173-4d2c-a70a-96b9b1b323b3/kube-multus/0.log"
Jul 09 19:23:48.595422 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:48.595316    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbvg5_a677a1da-0454-4e1e-9842-729623211e74/kube-multus-additional-cni-plugins/0.log"
Jul 09 19:23:48.623681 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:48.623655    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbvg5_a677a1da-0454-4e1e-9842-729623211e74/egress-router-binary-copy/0.log"
Jul 09 19:23:48.652570 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:48.652540    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbvg5_a677a1da-0454-4e1e-9842-729623211e74/cni-plugins/0.log"
Jul 09 19:23:48.679304 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:48.679276    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbvg5_a677a1da-0454-4e1e-9842-729623211e74/bond-cni-plugin/0.log"
Jul 09 19:23:48.705319 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:48.705288    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbvg5_a677a1da-0454-4e1e-9842-729623211e74/routeoverride-cni/0.log"
Jul 09 19:23:48.730423 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:48.730399    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbvg5_a677a1da-0454-4e1e-9842-729623211e74/whereabouts-cni-bincopy/0.log"
Jul 09 19:23:48.754248 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:48.754221    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cbvg5_a677a1da-0454-4e1e-9842-729623211e74/whereabouts-cni/0.log"
Jul 09 19:23:49.031262 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:49.031233    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5kshf_c7fde25b-60ca-405a-acf7-53e67418af9f/network-metrics-daemon/0.log"
Jul 09 19:23:49.056720 ip-10-0-140-224 kubenswrapper[2558]: I0709 19:23:49.056667    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5kshf_c7fde25b-60ca-405a-acf7-53e67418af9f/kube-rbac-proxy/0.log"